Knowledge distillation
Training a student model using behavioural information from a teacher model or an ensemble of models.
What does the smaller model learn from its teacher?
Knowledge distillation trains a target model using behavioural information from a teacher model or group of models. This can include outputs or probability distributions. The target, also called a student model, is often smaller and is intended to perform the required task with lower operating demands. What transfers is behaviour for the chosen training tasks; a complete, identical body of knowledge does not follow. Quantisation instead changes the numerical representation of model values.
In a fictional project, a teacher model is to produce short routing notes for service messages, stating the request and missing information. The notes are checked against the original messages before being used as training material. If the teacher invents information or overlooks an existing detail, the target model could inherit that error. Later assessment uses different messages excluded from training. Their expected results are defined for the business task. This allows reviewers to examine whether required content is preserved and missing information continues to be treated as missing.
Sources
Related terms
- Machine learningModels learn patterns from data and apply them to new inputs.
- Fine-tuningFine-tuning adapts a pretrained model through further training. Supplying current documents at runtime is a different operation.
- LLM evaluationLLM evaluation tests model outputs against defined tasks and criteria. One good example does not establish overall quality.
- Training dataThe examples from which a model learns relationships or parameters during training.
- Test data in machine learningData reserved for final assessment whose results were not used to select the model variant being assessed.
- Model quantisationRepresenting model weights and, depending on the method, intermediate values with lower numerical precision.
Your project with ASES
Describe your project to us. We will get back to you with an initial assessment.