Skip to content
Quantoros

How are reasoning models trained? What is RLVR?

LLM & AILevel: IntermediateTraining Methodology

How are reasoning models trained? What is RLVR?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

Reasoning models are typically trained via Reinforcement Learning with Verifiable Rewards (RLVR):

  1. Start with a capable base model (e.g. LLaMA, Qwen, or an internal model)
  2. Generate rollouts: model produces thinking trace + answer for many problems
  3. Verify correctness: for math problems, check final answer against ground truth. For code, run tests. No human labeler needed — the reward is binary (correct/incorrect).
  4. RL update (GRPO, PPO): reinforce trajectories that led to correct answers; penalize incorrect ones
  5. Emergent behavior: the model learns to use thinking tokens for complex problems, developing strategies like backtracking, self-checking, and decomposing

Why RLVR works: unlike RLHF which requires human preference labels, RLVR uses ground truth verifiers — math has objectively correct answers. This makes data collection cheap and scalable.

DeepSeek-R1 demonstrated that this approach works with open-source base models, achieving o1-comparable performance.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions