Skip to content
Quantoros

How do thinking token budgets work in production APIs?

LLM & AILevel: IntermediateThinking Tokens & Budgets

How do thinking token budgets work in production APIs?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

Major API providers expose thinking budget controls:

OpenAI (o-series): reasoning_effort parameter — low, medium, high. Maps to internal thinking token limits (~1K, ~10K, ~32K tokens). Higher effort = better quality, higher cost, higher latency.

Anthropic (Claude 3.7): thinking: {type: "enabled", budget_tokens: N} where N is max thinking tokens. Thinking tokens are billed at normal rates.

Practical guidance:

  • Use low/medium for most production queries — the marginal quality gain from high effort is often small for real-world tasks (not competition math)
  • Measure, don't assume — run your actual task distribution against different thinking budgets. The optimal setting is task-dependent.
  • Watch for billing surprises — on hidden-thinking APIs, you may be paying for 5,000 thinking tokens per query invisibly

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions