Reproducibility of AI outputs
Being able to repeat AI generation under documented conditions and compare results against defined criteria.
What is expected to stay the same in a repeat?
Reproducibility of AI outputs requires documented generation conditions. These include the model version, complete instructions and inputs, sources or tool results used, and output settings. A repeat can be assessed for identical text or for equivalent business results; these goals must be defined separately. A fixed seed can support comparison if the model offers it. It does not guarantee identical output, even together with low temperature.
In a fictional project, a model is to transfer an equipment identifier and report date from a service report into the fields asset_id and reported_date. Planned repeats use the same report, the same source versions and the same model configuration. Review criteria for correct field values, missing information and the expected format are defined beforehand. Different punctuation in an additional explanatory text may be irrelevant; a different equipment identifier would matter for the business task. Results are retained with their generation conditions so that later model or workflow changes can be compared systematically.
Sources
Related terms
- Prompt engineeringDeliberately designing and checking tasks, context and output rules for a generative model.
- LLM evaluationLLM evaluation tests model outputs against defined tasks and criteria. One good example does not establish overall quality.
- Test data in machine learningData reserved for final assessment whose results were not used to select the model variant being assessed.
- Temperature in language modelsA setting that influences the distribution of possible next tokens during sampling and therefore their selection.
- Top-p samplingSelecting the next token from a small set of likely candidates that together cover a probability threshold.
- Re-rankingRe-ranking reassesses search candidates already retrieved. Relevance to the query and the limits of the initial selection both matter.
Your project with ASES
Describe your project to us. We will get back to you with an initial assessment.