Skip to content
Quantoros

How does the KV cache shape production serving, and what two generation phases does it split into?

LLM & AILevel: AdvancedInference & Serving

How does the KV cache shape production serving, and what two generation phases does it split into?

Hints
  • Separate input preparation, prediction, queueing and data movement.
  • How do per-request latency, throughput and memory change with batch size?

Answer

This answer is part of mentoring

Advanced questions come with a full breakdown and a mock interview with a working quant researcher. The first call is free.

Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0

Related questions