Skip to content
Quantoros

How do you optimize prompts for cost and latency?

LLM & AILevel: IntermediateAdvanced Prompting

How do you optimize prompts for cost and latency?

Hints
  • Separate input preparation, prediction, queueing and data movement.
  • How do per-request latency, throughput and memory change with batch size?

Answer

Reduce input tokens:

  • Remove redundant instructions and examples
  • Compress few-shot examples (use tabular format, not verbose paragraphs)
  • Use prompt templates that inject only necessary context
  • Cache the system prompt with prompt caching (Anthropic, OpenAI) — reuse the KV cache

Reduce output tokens:

  • "Be concise. Answer in 2–3 sentences." in system prompt
  • Define output schema that eliminates preambles ("Sure! Here's the answer...")
  • Use structured output instead of natural language for machine-consumed outputs

Model routing:

  • Route simple queries to cheap/fast models (GPT-4o-mini, Haiku)
  • Route complex reasoning to expensive models (GPT-4, Claude Sonnet)

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions