Skip to content
Quantoros

What is the "scaling law" for test-time compute?

LLM & AILevel: IntermediateTest-Time Compute Scaling

What is the "scaling law" for test-time compute?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

OpenAI's work on o1 showed that reasoning model performance on benchmarks like AIME (math competition) scales roughly log-linearly with the number of thinking tokens:

  • More thinking tokens → better performance
  • Diminishing returns set in, but the curve is remarkably consistent
  • At some tasks (e.g. hard math), even 10,000 thinking tokens help vs. 100

The practical implication: you can tune thinking budget at inference time to control the quality/cost trade-off. This is exposed as the thinking_budget or reasoning_effort parameter in some APIs.

Important caveat: TTC scaling only works for tasks with verifiable structure — math, coding, logic puzzles. For open-ended creative tasks or tasks requiring world knowledge, more thinking doesn't reliably help.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions