LLM latency has suddenly increased from P95=2s to P95=15s. How do you debug?
ML System DesignLevel: IntermediateTroubleshooting Scenarios
LLM latency has suddenly increased from P95=2s to P95=15s. How do you debug?
Hints
- Separate input preparation, prediction, queueing and data movement.
- How do per-request latency, throughput and memory change with batch size?
Answer
- Check provider status: OpenAI/Anthropic status pages — are they experiencing degradation?
- TTFT vs total latency: is the first token slow (request queuing issue) or is generation slow (long output)?
- Trace analysis: check LLM tracing tool — is it LLM inference, retrieval, or pre/post-processing?
- Context length: has average input length increased? (new feature sending more context?)
- Model change: was there an automatic model routing change to a slower/larger model?
- Infrastructure: check GPU/CPU utilization on self-hosted models; check for connection pool exhaustion to the LLM API.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0