Skip to content
Quantoros

What are the problems with sigmoid as an activation function?

Deep LearningLevel: IntermediateNeural networks

What are the problems with sigmoid as an activation function?

Hints
  • Separate the linear score from the function mapping it to a probability.
  • Connect the target distribution to its negative log-likelihood.

Answer

The derivative of the sigmoid function for large positive or negative numbers is almost zero. From this comes the problem of vanishing gradient — during the backpropagation our net will not learn (or will learn drastically slow). One possible way to solve this problem is to use ReLU activation function.

Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0

Related questions