LLM evaluation
LLM evaluation tests model outputs against defined tasks and criteria. One good example does not establish overall quality.
Examine quality using defined cases
LLM evaluation systematically tests a language model's outputs against specific tasks and assessment criteria. It may examine factual correctness, support from supplied sources or compliance with expected boundaries.
In a fictional test, a model answers delivery questions. One case contains a confirmed date; another lacks it. In the second case, recognising missing information is part of the expected outcome. A fluently invented date would be a failure.
Valid paraphrases should not automatically count as wrong merely because they differ from the reference text. Conversely, linguistic similarity does not prove factual correctness. Record the model, inputs and configuration. Evaluating a limited test scope does not establish quality for every future case.
Sources
Related terms
- Large language modelA large language model processes language using learned patterns. Fluent text proves neither factual accuracy nor access to current business data.
- AI hallucinationAn AI hallucination is plausible-looking content that is factually false or invented. Fluent language is not evidence of correctness.
- Model cardA model card documents an AI model’s purpose, foundations, evaluation conditions and known limitations for a particular version.
- AI readinessAI readiness concerns the foundations of an AI project: objectives, usable data, skills, responsibilities and an operating workflow.
- Context engineeringContext engineering prepares information for individual model calls: relevant instructions, messages, sources and current states.
- Anomaly detectionMethods that flag unusual observations relative to a defined reference or a learned pattern.
Articles on this topic
AI & Automation with ASES
AI assistants and automated workflows