Why do we need activation functions?
Deep LearningLevel: FoundationNeural networks
Why do we need activation functions?
Hints
- Trace data and gradients through the model, noting shapes.
- Separate training from inference and identify a necessary assumption.
Answer
The main idea of using neural networks is to learn complex nonlinear functions. If we are not using an activation function in between different layers of a neural network, we are just stacking up multiple linear layers one on top of another and this leads to learning a linear function. The Nonlinearity comes only with the activation function, this is the reason we need activation functions.
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0