What is knowledge distillation and how is it used in LLM fine-tuning?
LLM & AILevel: IntermediateAdvanced Topics
What is knowledge distillation and how is it used in LLM fine-tuning?
Hints
- Which parameters change and which stay frozen?
- Separate memory for weights, gradients, optimizer states and activations.
Answer
Distillation trains a smaller student model to mimic the outputs of a larger teacher model:
Standard distillation (response distillation):
- Run teacher model on training inputs → collect outputs (or full probability distributions)
- Train student model to reproduce teacher outputs (KL divergence or cross-entropy on soft labels)
- Student benefits from teacher's "soft" probability distribution which contains more information than hard labels
Sequence-level KD for LLMs:
Teacher (GPT-4): given "Explain RAG" → generates high-quality explanation
Student (7B): trained to reproduce teacher's output on thousands of such examples
Used to create models like: Alpaca (LLaMA fine-tuned on GPT-3.5 outputs), WizardLM, Orca (reasoning steps from GPT-4).
Legal caveat: distillation from commercial models (OpenAI, Anthropic) may violate their terms of service. Check before using.
Reasoning model distillation: DeepSeek-R1 released distilled variants (8B, 14B, 32B) trained on R1's reasoning traces. The small models inherit long chain-of-thought reasoning capability from the 671B teacher — highly effective.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0