Plan how an AI pilot ends, too
A pilot needs criteria for continuing, narrowing its scope and ending. Functions already developed do not justify the next expansion on their own.
An AI pilot provides a basis for deciding on further use. That includes ending an unsuitable workflow. If only the next expansion stage is planned in advance, there is no clear way to handle results that argue against the intended use.
Define the required results before the trial
In this fictional example, a pilot is intended to pre-sort technical support requests. Critical portal incidents must not disappear into the routine queue without a visible flag. The team sets this boundary before testing and prepares suitable cases.
Microsoft recommends comparing AI configurations on the same cases and examining individual errors. Continuation therefore depends on whether the intended decisions and boundaries hold in the relevant cases. A convincing routine case does not answer the question of an overlooked critical report.
Allow more than one decision
The response depends on actual findings and an acceptable scope of use. The table describes possible decisions, not measured results:
Decision | Possible basis | Consequence |
|---|---|---|
Continue within limits | The necessary criteria are supported within the intended scope | Continue controlled trials of the checked scope |
Narrow the scope | One part is usable, but an important automated decision remains unresolved | Leave that part to a responsible person for now |
End the pilot | Essential requirements remain unmet for the intended use | Retire the automated function in an orderly way |
The NIST profile for generative AI provides for responsibilities and criteria for replacing, disengaging or deactivating unsuitable AI systems. It also recommends communication and alternative processes. This does not amount to a blanket economic judgement about a particular pilot. Deciding on its use requires the pilot’s own goals and results.
Make ending a handover
Stopping automatic pre-sorting must not leave received requests unattended. The team needs to find and continue handling open cases. Affected staff need to know which function ends and which workflow takes over.
Record the criteria, configuration examined and reason for the decision. Suitable test cases, identified limits or reusable integrations may remain useful after the pilot ends. Development work already invested does not replace a basis for the next expansion, however. Restarting is justified by a changed task or demonstrably improved results.
Sources
Read more
All articles
AI & Automation with ASES
AI assistants and automated workflows