How do you design for LLM API failures?
ML System DesignLevel: IntermediateScalability & Tradeoffs
How do you design for LLM API failures?
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
LLM APIs have higher failure rates than traditional APIs: rate limits, model availability, high latency spikes.
Resilience patterns:
- Multi-provider failover: if OpenAI fails, route to Anthropic; implement as a provider chain with health checks
- Circuit breaker: after N consecutive failures, short-circuit for T seconds; serve cached/fallback response
- Exponential backoff with jitter: retry with 2ˢ × random_jitter delay between attempts
- Timeout + degraded mode: if model doesn't respond in 10s, return a partial cached answer or "I'm having trouble right now"
- Queue-based buffering: for non-real-time tasks, put requests in a queue; process when LLM is available
- Rate limit budgeting: distribute rate limit tokens across tenants; shed load from low-priority users first
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0