How do thinking token budgets work in production APIs?
LLM & AILevel: IntermediateThinking Tokens & Budgets
How do thinking token budgets work in production APIs?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
Major API providers expose thinking budget controls:
OpenAI (o-series): reasoning_effort parameter — low, medium, high. Maps to internal thinking token limits (~1K, ~10K, ~32K tokens). Higher effort = better quality, higher cost, higher latency.
Anthropic (Claude 3.7): thinking: {type: "enabled", budget_tokens: N} where N is max thinking tokens. Thinking tokens are billed at normal rates.
Practical guidance:
- Use low/medium for most production queries — the marginal quality gain from high effort is often small for real-world tasks (not competition math)
- Measure, don't assume — run your actual task distribution against different thinking budgets. The optimal setting is task-dependent.
- Watch for billing surprises — on hidden-thinking APIs, you may be paying for 5,000 thinking tokens per query invisibly
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0