What is ReLU? How is it better than sigmoid or tanh?
Deep LearningLevel: IntermediateNeural networks
What is ReLU? How is it better than sigmoid or tanh?
Hints
- Separate the linear score from the function mapping it to a probability.
- Connect the target distribution to its negative log-likelihood.
Answer
ReLU is an abbreviation for Rectified Linear Unit. It is an activation function which has the value 0 for all negative values and the value f(x) = x for all positive values. The ReLU has a simple activation function which makes it fast to compute and while the sigmoid and tanh activation functions saturate at higher values, the ReLU has a potentially infinite activation, which addresses the problem of vanishing gradients.
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0