How are reasoning models trained? What is RLVR?
LLM & AILevel: IntermediateTraining Methodology
How are reasoning models trained? What is RLVR?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
Reasoning models are typically trained via Reinforcement Learning with Verifiable Rewards (RLVR):
- Start with a capable base model (e.g. LLaMA, Qwen, or an internal model)
- Generate rollouts: model produces thinking trace + answer for many problems
- Verify correctness: for math problems, check final answer against ground truth. For code, run tests. No human labeler needed — the reward is binary (correct/incorrect).
- RL update (GRPO, PPO): reinforce trajectories that led to correct answers; penalize incorrect ones
- Emergent behavior: the model learns to use thinking tokens for complex problems, developing strategies like backtracking, self-checking, and decomposing
Why RLVR works: unlike RLHF which requires human preference labels, RLVR uses ground truth verifiers — math has objectively correct answers. This makes data collection cheap and scalable.
DeepSeek-R1 demonstrated that this approach works with open-source base models, achieving o1-comparable performance.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0