Skip to content
Quantoros

Why is attention scaled by √dₖ?

LLM & AILevel: IntermediateAttention Mechanisms

Why is attention scaled by √dₖ?

Hints
  • Track Q, K and V shapes and which positions can interact.
  • Check masking, weight normalization and training versus autoregressive inference.

Answer

With large d_k, dot products grow large in magnitude, pushing softmax into saturation regions where gradients become vanishingly small. Dividing by √d_k keeps the variance of the dot product at ~1 regardless of d_k, maintaining healthy gradient flow.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions