Skip to content
Quantoros

What is RLHF?

LLM & AILevel: IntermediateTraining & Alignment

What is RLHF?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

Reinforcement Learning from Human Feedback — the core alignment technique:

  1. Collect human preference data: show annotators two model responses, they pick the better one
  2. Train a reward model on these comparisons
  3. Use PPO (Proximal Policy Optimization) to update the LLM toward responses with higher reward
  4. Add KL penalty to prevent the model from drifting too far from the original SFT model

Used by OpenAI for ChatGPT, Anthropic for Claude. DPO (Direct Preference Optimization) achieves similar results without the RL loop.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions