Skip to content
Quantoros

What is memory bandwidth and why does it matter for LLM inference?

ML System DesignLevel: IntermediateGPU Fundamentals

What is memory bandwidth and why does it matter for LLM inference?

Hints
  • Who owns the resource and when does its lifetime end?
  • Check copying, moving, exceptions and use after release.

Answer

LLM inference is memory-bandwidth bound (not compute-bound) for small batch sizes. Each forward pass requires loading all model weights from VRAM into compute units.

Why bandwidth matters:

  • 70B model in FP16: 140 GB of weights
  • A100 memory bandwidth: 2 TB/s
  • Minimum inference time per token: 140 GB / 2 TB/s = 70ms just from weight loading
  • H100 SXM5: 3.35 TB/s → 42ms minimum per token

Implication: faster memory bandwidth = lower latency for LLM inference. Compute (FLOPS) doesn't matter much for small batch autoregressive generation — you're limited by how fast you can read the weights.

At large batch sizes: compute-bound; FLOPS matter more. KV cache grows, memory becomes constrained differently.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions