What is the "scaling law" for test-time compute?
LLM & AILevel: IntermediateTest-Time Compute Scaling
What is the "scaling law" for test-time compute?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
OpenAI's work on o1 showed that reasoning model performance on benchmarks like AIME (math competition) scales roughly log-linearly with the number of thinking tokens:
- More thinking tokens → better performance
- Diminishing returns set in, but the curve is remarkably consistent
- At some tasks (e.g. hard math), even 10,000 thinking tokens help vs. 100
The practical implication: you can tune thinking budget at inference time to control the quality/cost trade-off. This is exposed as the thinking_budget or reasoning_effort parameter in some APIs.
Important caveat: TTC scaling only works for tasks with verifiable structure — math, coding, logic puzzles. For open-ended creative tasks or tasks requiring world knowledge, more thinking doesn't reliably help.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0