Skip to content
Quantoros

LLM latency has suddenly increased from P95=2s to P95=15s. How do you debug?

ML System DesignLevel: IntermediateTroubleshooting Scenarios

LLM latency has suddenly increased from P95=2s to P95=15s. How do you debug?

Hints
  • Separate input preparation, prediction, queueing and data movement.
  • How do per-request latency, throughput and memory change with batch size?

Answer

  1. Check provider status: OpenAI/Anthropic status pages — are they experiencing degradation?
  2. TTFT vs total latency: is the first token slow (request queuing issue) or is generation slow (long output)?
  3. Trace analysis: check LLM tracing tool — is it LLM inference, retrieval, or pre/post-processing?
  4. Context length: has average input length increased? (new feature sending more context?)
  5. Model change: was there an automatic model routing change to a slower/larger model?
  6. Infrastructure: check GPU/CPU utilization on self-hosted models; check for connection pool exhaustion to the LLM API.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions