Skip to content
Quantoros

In plain terms, what is knowledge distillation trying to achieve with a student and a teacher model?

LLM & AILevel: IntermediateFine-tuning & Alignment

In plain terms, what is knowledge distillation trying to achieve with a student and a teacher model?

Hints
  • Identify the stage: pretraining, adaptation, generation or evaluation.
  • Name the inputs, constraints and a case where the approach fails.

Answer

Distillation takes a large, capable but expensive teacher model and uses it to train a small student model. You run the frozen teacher over lots of inputs and train the student to reproduce its outputs.

Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0

Related questions