In plain terms, what is knowledge distillation trying to achieve with a student and a teacher model?
LLM & AILevel: IntermediateFine-tuning & Alignment
In plain terms, what is knowledge distillation trying to achieve with a student and a teacher model?
Hints
- Identify the stage: pretraining, adaptation, generation or evaluation.
- Name the inputs, constraints and a case where the approach fails.
Answer
Distillation takes a large, capable but expensive teacher model and uses it to train a small student model. You run the frozen teacher over lots of inputs and train the student to reproduce its outputs.
Source and licence: Skillumen — Awesome AI Engineer Interview Questions · CC0-1.0