What is memory bandwidth and why does it matter for LLM inference?
ML System DesignLevel: IntermediateGPU Fundamentals
What is memory bandwidth and why does it matter for LLM inference?
Hints
- Who owns the resource and when does its lifetime end?
- Check copying, moving, exceptions and use after release.
Answer
LLM inference is memory-bandwidth bound (not compute-bound) for small batch sizes. Each forward pass requires loading all model weights from VRAM into compute units.
Why bandwidth matters:
- 70B model in FP16: 140 GB of weights
- A100 memory bandwidth: 2 TB/s
- Minimum inference time per token: 140 GB / 2 TB/s = 70ms just from weight loading
- H100 SXM5: 3.35 TB/s → 42ms minimum per token
Implication: faster memory bandwidth = lower latency for LLM inference. Compute (FLOPS) doesn't matter much for small batch autoregressive generation — you're limited by how fast you can read the weights.
At large batch sizes: compute-bound; FLOPS matter more. KV cache grows, memory becomes constrained differently.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0