Skip to content
Quantoros

What is a Process Reward Model (PRM) and how does it differ from an Outcome Reward Model (ORM)?

LLM & AILevel: IntermediateProcess Reward Models

What is a Process Reward Model (PRM) and how does it differ from an Outcome Reward Model (ORM)?

Hints
  • Compare evaluating an intermediate reasoning step with evaluating only the final result.
  • What labels does each approach require, and how could a correct result hide an invalid step?

Answer

Both PRMs and ORMs are used to score reasoning model outputs, but they differ in granularity:

Outcome Reward Model (ORM):

  • Scores only the final answer: correct = 1, wrong = 0 (or soft reward)
  • Simple and cheap to train
  • Limited: provides no signal about which steps in the reasoning were good/bad

Process Reward Model (PRM):

  • Scores each intermediate reasoning step
  • Trained on human (or AI) labels of which steps are logically correct
  • Enables more fine-grained RL training and better search
Step 1: "Let x = speed of train A" → ✓ score: 0.95
Step 2: "Distance = speed × time, so x·t = 370" → ✓ score: 0.91
Step 3: "Therefore x = 370/4 = 92.5" → ✗ score: 0.2  (wrong operation)
Step 4: "So train A travels 92.5 mph" → ✗ score: 0.1

PRMs enable beam search at inference time: generate multiple reasoning paths, use the PRM to score partial paths, and prune bad branches early — before they lead to wrong answers.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions