What is Flash Attention?
LLM & AILevel: IntermediateAttention Mechanisms
What is Flash Attention?
Hints
- Track Q, K and V shapes and which positions can interact.
- Check masking, weight normalization and training versus autoregressive inference.
Answer
Standard attention materializes the full N×N attention matrix in GPU HBM, creating a memory and compute bottleneck. Flash Attention (Dao et al., 2022) is an IO-aware exact attention implementation:
- Tiles computation into blocks fitting in fast SRAM
- Uses online softmax to compute results in one pass
- Never materializes the full attention matrix
Result: 2–4× speedup, ~10× memory reduction, identical mathematical output (no approximation). Now standard in all production LLMs. Flash Attention 3 adds additional optimizations for H100 GPUs.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0