Skip to content
Quantoros

What role does an activation function play, and what happens to stacked layers without any non-linearity?

LLM & AILevel: FoundationTransformers & Architecture

What role does an activation function play, and what happens to stacked layers without any non-linearity?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

An activation function adds non-linearity between layers. Without it, stacking layers buys you nothing, because a chain of linear multiply-and-add layers collapses into a single linear layer, one straight line with no extra expressive power no matter how deep you go.

Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0

Related questions