How do you optimize LLM API costs in production?
ML System DesignLevel: IntermediateCost Optimization
How do you optimize LLM API costs in production?
Hints
- Start with quality, load and latency requirements.
- Trace data to prediction; identify a failure point and a validation method.
Answer
Input token reduction:
- Compress conversation history (summarize older turns)
- RAG chunk optimization: use fewer, more precise chunks
- Prompt optimization: remove verbose instructions that don't improve quality
Output token reduction:
- Set
max_tokenslimits explicitly - Instruct model to be concise in system prompt
- For structured outputs, JSON mode avoids verbose markdown explanations
Model routing:
- Route simple, low-risk queries → GPT-4o-mini (~15× cheaper than GPT-4o)
- Route complex reasoning, critical tasks → GPT-4o
- Use a fast classifier to route:
if complexity_score < 0.4: use cheap_model
Caching:
- Semantic cache (see above): 30–60% cost reduction on repetitive queries
- KV cache prefix caching: if your system prompt is always the same, API providers (Anthropic, OpenAI) cache KV state of repeated prefix
Quantified typical savings: model routing + semantic cache = 40–70% cost reduction for most production chatbots.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0