What is DPO (Direct Preference Optimization)?
LLM & AILevel: IntermediateAlignment Training
What is DPO (Direct Preference Optimization)?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
DPO (Rafailov et al., 2023) achieves RLHF-equivalent alignment without the RL loop. It directly trains on preference pairs using a closed-form loss derived from the reward model:
Where y_w = preferred response, y_l = rejected response, π_ref = reference model.
Why DPO won: simpler (no reward model, no PPO), more stable training, competitive quality. Now the dominant alignment method for open-source models. ORPO, SimPO, and other variants further simplify by removing the reference model.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0