Skip to content
Quantoros

What is continual pre-training vs fine-tuning?

LLM & AILevel: IntermediateFundamentals

What is continual pre-training vs fine-tuning?

Hints
  • Which parameters change and which stay frozen?
  • Separate memory for weights, gradients, optimizer states and activations.

Answer

Continual Pre-TrainingSupervised Fine-Tuning (SFT)
ObjectiveNext-token prediction on domain textSupervised on instruction-response pairs
DataRaw domain text (papers, code, docs)Curated Q&A or instruction-following pairs
GoalInject domain knowledge into weightsTeach behavior / format / task patterns
When to useModel lacks domain vocabulary or conceptsModel has knowledge, needs to follow instructions

Typical workflow: pre-train on domain corpus → SFT → alignment (RLHF/DPO).

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions