Skip to content
Quantoros

What is Flash Attention?

LLM & AILevel: IntermediateAttention Mechanisms

What is Flash Attention?

Hints
  • Track Q, K and V shapes and which positions can interact.
  • Check masking, weight normalization and training versus autoregressive inference.

Answer

Standard attention materializes the full N×N attention matrix in GPU HBM, creating a memory and compute bottleneck. Flash Attention (Dao et al., 2022) is an IO-aware exact attention implementation:

  1. Tiles computation into blocks fitting in fast SRAM
  2. Uses online softmax to compute results in one pass
  3. Never materializes the full attention matrix

Result: 2–4× speedup, ~10× memory reduction, identical mathematical output (no approximation). Now standard in all production LLMs. Flash Attention 3 adds additional optimizations for H100 GPUs.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions