Infrastructure & Deployment

Distillation (Knowledge Distillation)

Training a smaller model to mimic a larger, more capable one.

Definition

Knowledge distillation is a training technique where a smaller 'student' model is trained to reproduce the outputs of a larger 'teacher' model. The student model learns not just from correct answers but from the full probability distribution of the teacher's responses, which contains richer information than simple correct/incorrect labels. The result is a small model that approaches the performance of the large model at a fraction of the size and running cost.

Related Terms

Model Compression

Techniques to make large models smaller so they run on less powerful hardware.

Fine-tuning

Further training a pre-trained model on specific data to specialise it for a task.

Quantisation

Reducing a model's numerical precision to make it smaller and faster, with minimal quality loss.

Heard enough terminology — ready to talk outcomes?

We translate AI concepts into measurable business results. No upfront fees — you pay only when independently verified results are delivered.

← Back to glossary

Disclaimer

This definition is provided for educational and informational purposes only. It represents a general explanation of a technical concept and does not constitute professional, technical, or investment advice. Artificial intelligence is a rapidly evolving field; terminology, techniques, and capabilities change frequently. Coaley Peak Ltd makes no warranty as to the accuracy, completeness, or currency of the information provided. Nothing on this page should be relied upon as the sole basis for commercial, technical, legal, or investment decisions without independent professional advice.

Document reference: ISO_webpage_knowledge-base_glossary_v1

Last modified: 29 March 2026

Knowledge Base·Infrastructure & Deployment·Distillation (Knowledge Distillation)