Skip to content
Quantoros

How do you design for LLM API failures?

ML System DesignLevel: IntermediateScalability & Tradeoffs

How do you design for LLM API failures?

Hints
  • Start with quality, load and latency requirements.
  • Trace data to prediction; identify a failure point and a validation method.

Answer

LLM APIs have higher failure rates than traditional APIs: rate limits, model availability, high latency spikes.

Resilience patterns:

  1. Multi-provider failover: if OpenAI fails, route to Anthropic; implement as a provider chain with health checks
  2. Circuit breaker: after N consecutive failures, short-circuit for T seconds; serve cached/fallback response
  3. Exponential backoff with jitter: retry with 2ˢ × random_jitter delay between attempts
  4. Timeout + degraded mode: if model doesn't respond in 10s, return a partial cached answer or "I'm having trouble right now"
  5. Queue-based buffering: for non-real-time tasks, put requests in a queue; process when LLM is available
  6. Rate limit budgeting: distribute rate limit tokens across tenants; shed load from low-priority users first

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions