Your AI agent burns too many tokens per task. How do you reduce token consumption?
LLM & AILevel: IntermediateTroubleshooting Scenarios
Your AI agent burns too many tokens per task. How do you reduce token consumption?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
- Compress tool observations: truncate or summarize long tool outputs before adding to context. A 10,000-token search result → 500-token summary.
- Prune conversation history: keep last K turns + initial task description; drop intermediate reasoning steps once a subtask is complete.
- Use smaller models for subtasks: route simple tool-call decisions to a fast/cheap model; reserve expensive models for complex reasoning.
- Parallel tool calls: if multiple independent information lookups are needed, batch them in one model call rather than N sequential calls.
- Plan first, then execute: a good upfront plan reduces wasted exploratory steps.
- Cache tool results: if the agent might call the same tool with the same parameters multiple times, cache results within a session.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0