What benchmarks are used to evaluate reasoning models?
LLM & AILevel: IntermediateEvaluation & Benchmarks
What benchmarks are used to evaluate reasoning models?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
| Benchmark | Domain | Why It's Used |
|---|---|---|
| AIME | Math (competition) | Hard problems requiring multi-step exact reasoning; o1 was first model to score meaningfully |
| MATH-500 | Math (diverse) | Broad math benchmark; saturating for best models |
| AMC 2023 | Math (competition) | Similar to AIME, slightly easier |
| HumanEval / LiveCodeBench | Coding | Code generation and debugging |
| SWE-bench | Software engineering | Real GitHub issues; requires understanding codebases |
| GPQA Diamond | PhD-level science | Expert-level questions in biology, chemistry, physics |
| ARC-AGI | Abstract reasoning | Visual pattern completion; tests novel reasoning |
| FrontierMath | Advanced math | New dataset; problems that require novel approaches |
Warning for practitioners: all benchmarks saturate over time and can be gamed. LLM providers optimize for benchmark scores, sometimes at the expense of general capability. Always evaluate on your own held-out distribution.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0