Skip to content
Quantoros

What is KV Cache and how does it speed up inference?

LLM & AILevel: FoundationInference & Decoding

What is KV Cache and how does it speed up inference?

Hints
  • Separate input preparation, prediction, queueing and data movement.
  • How do per-request latency, throughput and memory change with batch size?

Answer

At each attention layer, the model computes K and V matrices from all previous tokens. Without caching, these are recomputed from scratch at every generation step — O(N²) total computation for N tokens.

KV Cache stores these K and V tensors. Each new step only computes the new token's Q, K, V, appends to cache, and reuses all previous K,V. Result: O(N) total computation instead of O(N²).

Memory growth: KV cache size = 2 × layers × heads × d_k × sequence_length × batch_size × bytes_per_element. For large batches or long sequences this dominates GPU memory — addressed by GQA, PagedAttention, KV quantization.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions