Why is attention scaled by √dₖ?
LLM & AILevel: IntermediateAttention Mechanisms
Why is attention scaled by √dₖ?
Hints
- Track Q, K and V shapes and which positions can interact.
- Check masking, weight normalization and training versus autoregressive inference.
Answer
With large d_k, dot products grow large in magnitude, pushing softmax into saturation regions where gradients become vanishingly small. Dividing by √d_k keeps the variance of the dot product at ~1 regardless of d_k, maintaining healthy gradient flow.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0