What are the problems with sigmoid as an activation function?
Deep LearningLevel: IntermediateNeural networks
What are the problems with sigmoid as an activation function?
Hints
- Separate the linear score from the function mapping it to a probability.
- Connect the target distribution to its negative log-likelihood.
Answer
The derivative of the sigmoid function for large positive or negative numbers is almost zero. From this comes the problem of vanishing gradient — during the backpropagation our net will not learn (or will learn drastically slow). One possible way to solve this problem is to use ReLU activation function.
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0