Assess language quality and task accuracy separately

An easy-to-read translation can contain an incorrect work instruction. Multilingual automation therefore needs separate assessments of language quality and task correctness. A polite tone or an understandable sentence is insufficient approval for the next action.

A fluent sentence can name the wrong target

In this fictional example, a German instruction is translated for an English-speaking support team. The source text is: Nur die Testumgebung neu starten. Die Produktivumgebung unverändert lassen. A deliberately incorrect English example reads: Restart the production environment. This is a constructed failure case.

The English sentence is understandable. In meaning, it changes the target from the test environment to production and omits the explicit restriction. A review focused only on fluency would not adequately examine the choice of target environment.

The team therefore records two judgements: Is the text usable in the target language, and does it preserve the intended instruction, including the environment and restriction? Before execution, the action the application derives from it and whether that action is permitted also need checking.

A language identifier has a different job

Microsoft describes language detection as identifying a document’s predominant language with an associated confidence score. That score concerns language detection. It proves neither a correct translation nor an appropriate restart task.

A successful translation can also be mapped incorrectly later. If the translated text still specifies the test environment while the application selects another target, the discrepancy lies in a later processing step. Review therefore needs visibility into both text and derived action.

Review cases by language and error type

Microsoft recommends comparing AI evaluations on the same cases and examining individual errors. For this workflow, cases should also include negations, restrictions and differently named environments. The expected work instruction remains recorded so it can be checked.

Keep results separate by language and case type. Otherwise, a good overall rating does not show which language loses an important restriction. Involve people who can assess the relevant language and workflow in the linguistic and task review.

Approval needs suitable results for the intended use. Well-written texts are one part of that; preserved meaning and the checked subsequent action determine whether the automation fulfils the task.

Sources

• Microsoft — How to perform language detection

• Microsoft — Evaluate models, prompts, and agents