Skip to content
Quantoros

What is the KV cache and what wasteful work does it eliminate?

LLM & AILevel: FoundationInference & Serving

What is the KV cache and what wasteful work does it eliminate?

Hints
  • Separate input preparation, prediction, queueing and data movement.
  • How do per-request latency, throughput and memory change with batch size?

Answer

The KV cache stores the Key and Value vectors of every token the model has already seen. To generate the next token, attention has to look back at the K and V of all earlier tokens, and those never change once computed. Without a cache the model recomputes all of them from scratch every single step, which is pure repeated arithmetic.

Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0

Related questions