How do you optimize prompts for cost and latency?
LLM & AILevel: IntermediateAdvanced Prompting
How do you optimize prompts for cost and latency?
Hints
- Separate input preparation, prediction, queueing and data movement.
- How do per-request latency, throughput and memory change with batch size?
Answer
Reduce input tokens:
- Remove redundant instructions and examples
- Compress few-shot examples (use tabular format, not verbose paragraphs)
- Use prompt templates that inject only necessary context
- Cache the system prompt with prompt caching (Anthropic, OpenAI) — reuse the KV cache
Reduce output tokens:
- "Be concise. Answer in 2–3 sentences." in system prompt
- Define output schema that eliminates preambles ("Sure! Here's the answer...")
- Use structured output instead of natural language for machine-consumed outputs
Model routing:
- Route simple queries to cheap/fast models (GPT-4o-mini, Haiku)
- Route complex reasoning to expensive models (GPT-4, Claude Sonnet)
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0