What role does an activation function play, and what happens to stacked layers without any non-linearity?
LLM & AILevel: FoundationTransformers & Architecture
What role does an activation function play, and what happens to stacked layers without any non-linearity?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
An activation function adds non-linearity between layers. Without it, stacking layers buys you nothing, because a chain of linear multiply-and-add layers collapses into a single linear layer, one straight line with no extra expressive power no matter how deep you go.
Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0