What is causal masking?
LLM & AILevel: FoundationAttention Mechanisms
What is causal masking?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
Causal masking ensures the model cannot "see" future tokens during generation — enforcing the autoregressive property. Implemented by adding a mask matrix before softmax: future positions get -∞ (which becomes 0 after softmax), so token i only attends to tokens 0..i.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0