NeoMundi publishes a first study on the stability and reproducibility of its runtime signals

Fatima Ezzahrae Gouarab, Data Scientist, statistician and scientific contributor to the NeoMundi Research Observatory, is conducting work on the stability and reproducibility of the signals produced by NeoMundi ControlTower.

DOI : https://doi.org/10.5281/zenodo.21499715

This first experimental phase is based on 680 generations, carried out with three model providers, two generation temperatures and several corpora covering factual questions, reasoning tasks, speculative situations and stress-test scenarios. The objective is to determine whether NeoMundi signals maintain a consistent reading when experimental conditions vary.

Context and starting point

This study follows on from earlier work on the actionability of governance signals. During new executions of the initial protocol, an unexpected inversion of the distribution of ALLOW and FLAG decisions was observed, with no identified change to the protocol or to the NeoMundi API version.

This observation, comparable to the silent regime changes already documented in the NeoMundi Barometers, motivated the progressive construction of four successive experimental campaigns.

Key results

The campaigns describe signal distributions in the configurations tested:

• On the classical corpora, FLAG decision rates remain low and close across the different configurations, between 0% and 4%, despite variations in provider and temperature.
• On the stress-test corpus, this rate reaches 11.7% and is mainly concentrated on situations exposed to verifiable factual uncertainty:
  – 33.3% for current-affairs questions
  – 16.7% for false premises
  – 12.5% for fictional entities or requests for precise figures

In the configurations studied, the published results do not show a determining effect of temperature; this observation does not generalize to other tasks or models.

Across the 480 generations of the two main campaigns, the report describes a systematic correspondence between DROP and FLAG. Whether this follows from the calculation rule remains to be established; by itself, it is not independent validation. Stability scores clearly distinguish ALLOW cases from FLAG cases. This distinction between signal stability and other dimensions (notably factual coherence) is at the heart of governance metrology (see also our reference article Stability vs factual coherence in production AI ).

An observation that continues to be monitored

During this research, the repetition of an initial configuration also revealed a significant change in the decision regime. This observation remains unexplained at this stage and is now part of the phenomena monitored longitudinally.

These campaigns document regularities and an unexplained regime change in the configurations studied. They do not establish the instrument’s repeatability on strictly identical input. That property requires dedicated tests with a fixed prompt, response, evaluators and configuration.

Next steps

Fatima’s work is now continuing with a larger number of observations in order to test the statistical robustness of these initial trends and to broaden the range of configurations studied.

This first phase provides observations for the next tests: repeatability on fixed input, sensitivity to controlled variations, and the usefulness of signals for independent analysis of the responses.

Download the full note

Stability and Reproducibility of the NeoMundi ControlTower Runtime Signal

Comparative multi-provider, multi-corpus and multi-configuration analysis

Author of the report