Skip to content
Quantoros

How do you optimize LLM API costs in production?

ML System DesignLevel: IntermediateCost Optimization

How do you optimize LLM API costs in production?

Hints
  • Start with quality, load and latency requirements.
  • Trace data to prediction; identify a failure point and a validation method.

Answer

Input token reduction:

  • Compress conversation history (summarize older turns)
  • RAG chunk optimization: use fewer, more precise chunks
  • Prompt optimization: remove verbose instructions that don't improve quality

Output token reduction:

  • Set max_tokens limits explicitly
  • Instruct model to be concise in system prompt
  • For structured outputs, JSON mode avoids verbose markdown explanations

Model routing:

  • Route simple, low-risk queries → GPT-4o-mini (~15× cheaper than GPT-4o)
  • Route complex reasoning, critical tasks → GPT-4o
  • Use a fast classifier to route: if complexity_score < 0.4: use cheap_model

Caching:

  • Semantic cache (see above): 30–60% cost reduction on repetitive queries
  • KV cache prefix caching: if your system prompt is always the same, API providers (Anthropic, OpenAI) cache KV state of repeated prefix

Quantified typical savings: model routing + semantic cache = 40–70% cost reduction for most production chatbots.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions