Knowledge Distillation

Knowledge distillation trains a small model to imitate a large one. The small model learns from the large model's outputs rather than only from the original labels.

The insight is that the teacher's full output distribution carries more information than a hard label. Knowing an image is 70 percent cat and 25 percent fox teaches more than knowing it is a cat.

Think of it like this. Think of an expert who does not just mark answers right or wrong, but says how confident they are and what the plausible alternatives were. That extra signal is what makes the student learn faster.

How it works

• Train a large teacher model first, or take an existing one.
• The student learns from the teacher's outputs, the full probability distribution rather than only the correct answer.
• Temperature softens the distribution, revealing the teacher's relative confidence across classes instead of a single spike.
• Often combined with the real labels, balancing imitation against ground truth.

Trade-offs

• Much cheaper to run, often a fraction of the size at a large share of the quality.
• You need the teacher first, so total training cost goes up, not down.
• The student inherits the teacher's mistakes, including its biases.
• There is a floor. A student far smaller than its teacher cannot represent everything it knows.

Common mistakes

• "It is the same as quantization." Distillation trains a new, smaller model. Quantization stores the same model less precisely. They stack.
• "The student matches the teacher." It approximates. The gap widens as the size difference grows.
• "Only the final answer matters." The soft distribution is the whole point. Training on hard labels alone is just ordinary training on a smaller model.