AI demos show behaviour under selected conditions
A demo helps assess an idea. Later deployment also depends on the cases missing from the presentation.
Examine selection before results
A successful AI presentation can show that a task works under the conditions demonstrated. To make a claim about later operation, the team also needs to examine which inputs, ambiguities and boundaries were covered. The NIST AI RMF recommends documenting performance under deployment-like conditions and limits to generalisation.
For a support pilot, for example, distinguish between cases containing clear fault reports and those also containing mixed requests. A clear formulation calls for a different decision from a message whose meaning first needs clarification. A good result on one case type does not answer the question about the other.
Compare two case lists with different purposes
A fictional software company compares two planned selections. Clear messages in the demo are intended to make the workflow understandable. The later assessment set deliberately includes additional difficulties.
Case type | Demo selection | Planned assessment set |
|---|---|---|
Clearly described software fault | 8 | 16 |
Unambiguous access request | 4 | 14 |
Unclear or mixed message | 0 | 12 |
Message outside the task | 0 | 8 |
The totals are 12 and 50. These numbers describe our invented selection; success rates and operational frequencies would need separate measurement. The empty demo groups make visible which questions the presentation leaves open.
The larger set is also assembled for a purpose. Deliberately including difficult cases does not make their share automatically reflect daily incoming messages. That is useful for finding faults. Estimating the operational error proportion would require selection and analysis aligned with its actual distribution.
Use the demo for the right decision
If the team initially needs to understand a narrowly scoped workflow and its interface, a demo can be a useful feasibility exercise. If only clearly defined inputs belong to that first scope, the presentation need not already cover every imaginable later extension. The next deployment step must then respect that limited scope.
Examples repeatedly used to improve prompts subsequently help detect recurrence of known errors. An independent assessment of new cases needs additional reserved examples. Google describes how repeated decisions based on the same test data change its evidential value in machine learning. Applied to prompt selection, the demo remains useful, but its known cases do not also act as an independent final check.
Sources
Read more
All articles
AI & Automation with ASES
AI assistants and automated workflows