Skip to content
Quantoros

What is knowledge distillation and how is it used in LLM fine-tuning?

LLM & AILevel: IntermediateAdvanced Topics

What is knowledge distillation and how is it used in LLM fine-tuning?

Hints
  • Which parameters change and which stay frozen?
  • Separate memory for weights, gradients, optimizer states and activations.

Answer

Distillation trains a smaller student model to mimic the outputs of a larger teacher model:

Standard distillation (response distillation):

  • Run teacher model on training inputs → collect outputs (or full probability distributions)
  • Train student model to reproduce teacher outputs (KL divergence or cross-entropy on soft labels)
  • Student benefits from teacher's "soft" probability distribution which contains more information than hard labels

Sequence-level KD for LLMs:

Teacher (GPT-4): given "Explain RAG" → generates high-quality explanation
Student (7B): trained to reproduce teacher's output on thousands of such examples

Used to create models like: Alpaca (LLaMA fine-tuned on GPT-3.5 outputs), WizardLM, Orca (reasoning steps from GPT-4).

Legal caveat: distillation from commercial models (OpenAI, Anthropic) may violate their terms of service. Check before using.

Reasoning model distillation: DeepSeek-R1 released distilled variants (8B, 14B, 32B) trained on R1's reasoning traces. The small models inherit long chain-of-thought reasoning capability from the 671B teacher — highly effective.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions