Skip to content
Quantoros

What is DPO (Direct Preference Optimization)?

LLM & AILevel: IntermediateAlignment Training

What is DPO (Direct Preference Optimization)?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

DPO (Rafailov et al., 2023) achieves RLHF-equivalent alignment without the RL loop. It directly trains on preference pairs using a closed-form loss derived from the reward model:

LDPO=−E[log⁡σ(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))]\mathcal{L}_{DPO} = -\mathbb{E}[\log \sigma(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)})]

Where y_w = preferred response, y_l = rejected response, π_ref = reference model.

Why DPO won: simpler (no reward model, no PPO), more stable training, competitive quality. Now the dominant alignment method for open-source models. ORPO, SimPO, and other variants further simplify by removing the reference model.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions