How does the KV cache shape production serving, and what two generation phases does it split into?
LLM & AILevel: AdvancedInference & Serving
How does the KV cache shape production serving, and what two generation phases does it split into?
Hints
- Separate input preparation, prediction, queueing and data movement.
- How do per-request latency, throughput and memory change with batch size?
Answer
This answer is part of mentoring
Advanced questions come with a full breakdown and a mock interview with a working quant researcher. The first call is free.
Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0