Skip to content
Quantoros

What is ReLU? How is it better than sigmoid or tanh?

Deep LearningLevel: IntermediateNeural networks

What is ReLU? How is it better than sigmoid or tanh?

Hints
  • Separate the linear score from the function mapping it to a probability.
  • Connect the target distribution to its negative log-likelihood.

Answer

ReLU is an abbreviation for Rectified Linear Unit. It is an activation function which has the value 0 for all negative values and the value f(x) = x for all positive values. The ReLU has a simple activation function which makes it fast to compute and while the sigmoid and tanh activation functions saturate at higher values, the ReLU has a potentially infinite activation, which addresses the problem of vanishing gradients.

Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0

Related questions