What is continual pre-training vs fine-tuning?
LLM & AILevel: IntermediateFundamentals
What is continual pre-training vs fine-tuning?
Hints
- Which parameters change and which stay frozen?
- Separate memory for weights, gradients, optimizer states and activations.
Answer
| Continual Pre-Training | Supervised Fine-Tuning (SFT) | |
|---|---|---|
| Objective | Next-token prediction on domain text | Supervised on instruction-response pairs |
| Data | Raw domain text (papers, code, docs) | Curated Q&A or instruction-following pairs |
| Goal | Inject domain knowledge into weights | Teach behavior / format / task patterns |
| When to use | Model lacks domain vocabulary or concepts | Model has knowledge, needs to follow instructions |
Typical workflow: pre-train on domain corpus → SFT → alignment (RLHF/DPO).
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0