What is RLHF?
LLM & AILevel: IntermediateTraining & Alignment
What is RLHF?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
Reinforcement Learning from Human Feedback — the core alignment technique:
- Collect human preference data: show annotators two model responses, they pick the better one
- Train a reward model on these comparisons
- Use PPO (Proximal Policy Optimization) to update the LLM toward responses with higher reward
- Add KL penalty to prevent the model from drifting too far from the original SFT model
Used by OpenAI for ChatGPT, Anthropic for Claude. DPO (Direct Preference Optimization) achieves similar results without the RL loop.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0