NeoMundi publishes a first study on the stability and reproducibility of its runtime signals

Fatima Ezzahrae Gouarab, Data Scientist, statistician and scientific contributor to the NeoMundi Research Observatory, is conducting work on the stability and reproducibility of the signals produced by NeoMundi ControlTower.

DOI : https://doi.org/10.5281/zenodo.21499715

This first experimental phase is based on 680 generations, carried out with three model providers, two generation temperatures and several corpora covering factual questions, reasoning tasks, speculative situations and stress-test scenarios. The objective is to determine whether NeoMundi signals maintain a consistent reading when experimental conditions vary.

Context and starting point

This study follows on from earlier work on the actionability of governance signals. During new executions of the initial protocol, an unexpected inversion of the distribution of ALLOW and FLAG decisions was observed, with no identified change to the protocol or to the NeoMundi API version.

This observation, comparable to the silent regime changes already documented in the NeoMundi Barometers, motivated the progressive construction of four successive experimental campaigns.

Key results

The results obtained provide several elements supporting the reproducibility hypothesis:

• On the classical corpora, FLAG decision rates remain low and close across the different configurations, between 0% and 4%, despite variations in provider and temperature.
• On the stress-test corpus, this rate reaches 11.7% and is mainly concentrated on situations exposed to verifiable factual uncertainty:
  – 33.3% for current-affairs questions
  – 16.7% for false premises
  – 12.5% for fictional entities or requests for precise figures

Temperature does not appear to be a determining factor in the configurations studied.

The correspondence between the DROP profile and the FLAG decision remains systematic (perfect bijection) across the 480 generations of the two main campaigns. Stability scores clearly distinguish ALLOW cases from FLAG cases. This distinction between signal stability and other dimensions (notably factual coherence) is at the heart of governance metrology (see also our reference article Stability vs factual coherence in production AI).

An observation that continues to be monitored

During this research, the repetition of an initial configuration also revealed a significant change in the decision regime. This observation remains unexplained at this stage and is now part of the phenomena monitored longitudinally.

At this stage, none of the observed elements invalidate the hypothesis of the reproducibility of the NeoMundi signal under the tested conditions. On the contrary, the results strengthen its plausibility through their consistency across campaigns, their structured sensitivity to the situations submitted, and the alignment of the various metrics produced.

Next steps

Fatima’s work is now continuing with a larger number of observations in order to test the statistical robustness of these initial trends and to broaden the range of configurations studied.

This first phase does not conclude the demonstration, but it establishes that, at this stage, no result invalidates the hypothesis of the reproducibility of the NeoMundi signal; several converging observations instead consolidate it.

Download the full note

Stability and Reproducibility of the NeoMundi ControlTower Runtime Signal

Comparative multi-provider, multi-corpus and multi-configuration analysis

Author of the report