Model quantisation
Representing model weights and, depending on the method, intermediate values with lower numerical precision.
Which model values use a smaller representation?
Model quantisation represents weights and, in some methods, intermediate values with fewer bits. Model data can therefore require less memory while the number of parameters stays the same. The coarser representation can change results. Whether calculations become faster also depends on hardware, runtime and method. A smaller model file therefore does not describe the total memory requirement of a running application.
A fictional team plans to summarise service reports locally for an internal overview. It wants to compare a version with 16-bit weights against a quantised 8-bit version of the same base model. Instructions and approved report texts stay the same for the planned comparison. Reviewers examine omitted work steps and invented additions, alongside actual memory use and response time on the intended computer. Only this combined assessment shows whether the chosen variant performs the task with the available resources.
Sources
Related terms
- Machine learningModels learn patterns from data and apply them to new inputs.
- AI inferenceApplying an already trained model to inputs to obtain a result.
- LLM evaluationLLM evaluation tests model outputs against defined tasks and criteria. One good example does not establish overall quality.
- Test data in machine learningData reserved for final assessment whose results were not used to select the model variant being assessed.
- Knowledge distillationTraining a student model using behavioural information from a teacher model or an ensemble of models.
- Re-rankingRe-ranking reassesses search candidates already retrieved. Relevance to the query and the limits of the initial selection both matter.
Your project with ASES
Describe your project to us. We will get back to you with an initial assessment.