What is a Process Reward Model (PRM) and how does it differ from an Outcome Reward Model (ORM)?
LLM & AILevel: IntermediateProcess Reward Models
What is a Process Reward Model (PRM) and how does it differ from an Outcome Reward Model (ORM)?
Hints
- Compare evaluating an intermediate reasoning step with evaluating only the final result.
- What labels does each approach require, and how could a correct result hide an invalid step?
Answer
Both PRMs and ORMs are used to score reasoning model outputs, but they differ in granularity:
Outcome Reward Model (ORM):
- Scores only the final answer: correct = 1, wrong = 0 (or soft reward)
- Simple and cheap to train
- Limited: provides no signal about which steps in the reasoning were good/bad
Process Reward Model (PRM):
- Scores each intermediate reasoning step
- Trained on human (or AI) labels of which steps are logically correct
- Enables more fine-grained RL training and better search
Step 1: "Let x = speed of train A" → ✓ score: 0.95
Step 2: "Distance = speed × time, so x·t = 370" → ✓ score: 0.91
Step 3: "Therefore x = 370/4 = 92.5" → ✗ score: 0.2 (wrong operation)
Step 4: "So train A travels 92.5 mph" → ✗ score: 0.1
PRMs enable beam search at inference time: generate multiple reasoning paths, use the PRM to score partial paths, and prune bad branches early — before they lead to wrong answers.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0