Language model token
Tokens are processing units for language models. They do not reliably correspond to whole words and differ from design tokens.
Measure text in processing units
A token is a unit into which a language model divides text for processing. Depending on the tokeniser, it may correspond to a word, part of a word or a character. The count depends on the method and the actual text.
In a fictional project, the same product catalogue will be processed in German and Turkish. The same number of entries does not automatically mean the same token count. Check the actual texts with the appropriate tokeniser when estimating input limits or usage.
A fixed conversion from words to tokens is unreliable for this purpose. A design token has a different meaning: it names a design value such as a colour. These concepts share a name but serve different purposes.
Sources
Related terms
- Context windowThe context window limits how many tokens a language model can consider in one processing step. It is not a permanent knowledge archive.
- Design tokenA design token stores a named design decision. Multiple components can use it, allowing a value to be changed in a defined place.
- Model cardA model card documents an AI model’s purpose, foundations, evaluation conditions and known limitations for a particular version.
- AI readinessAI readiness concerns the foundations of an AI project: objectives, usable data, skills, responsibilities and an operating workflow.
- Context engineeringContext engineering prepares information for individual model calls: relevant instructions, messages, sources and current states.
- Anomaly detectionMethods that flag unusual observations relative to a defined reference or a learned pattern.
Articles on this topic
AI & Automation with ASES
AI assistants and automated workflows