What is the LLM training pipeline?
LLM & AILevel: IntermediateTraining & Alignment
What is the LLM training pipeline?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
- Pre-training — next-token prediction on trillions of web/book/code tokens. Builds general knowledge and language understanding.
- SFT (Supervised Fine-Tuning) — fine-tune on curated instruction-following examples. Teaches the model to follow instructions.
- Reward Modeling — train a reward model on human preference pairs (A vs B, which is better?)
- RLHF/DPO — optimize model outputs toward higher reward. Creates the helpful, harmless, honest behavior of deployed models.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0