Skip to content
Quantoros

What is causal masking?

LLM & AILevel: FoundationAttention Mechanisms

What is causal masking?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

Causal masking ensures the model cannot "see" future tokens during generation — enforcing the autoregressive property. Implemented by adding a mask matrix before softmax: future positions get -∞ (which becomes 0 after softmax), so token i only attends to tokens 0..i.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions