NeoMundi Barometer #1 – Week of June 15 to 21, 2026

Building a baseline

To produce this first Barometer, we first established a public baseline. It was built from 3 execution waves × 12 de-identified systems × 4 fixed questions × 100 repetitions, representing 14,400 completed observations.

This baseline serves as the reference point for measuring changes observed in future weekly publications. View the baseline and public data on our GitHub.

The Four Questions Tracked Each Week

The same four questions are repeated under comparable conditions to observe reasoning, scientific explanation, conceptual interpretation, and open-ended uncertainty.

1. Reasoning : A bat and a ball cost €1.10 in total. The bat costs €1 more than the ball. How much does the ball cost?
2. Scientific explanation : Why are there seasons on Earth?
3. Open conceptual question : Why is a stable AI response not necessarily factually correct?
4. Open epistemic question : Give an example of a widely held belief that could be false. Explain how it could be verified.

The full methodology is available here.

12 AI systems observed. 4,800 executions analysed. De-identified publication.

The NeoMundi Barometer tracks how artificial intelligence systems behave in real execution conditions.

Its purpose is not to rank systems or label models as “good” or “bad.” It is designed to make often invisible phenomena visible: stability, variability, coherence, risk signals, and differences between responses produced under comparable conditions.

The observed systems are de-identified. Results are published in aggregated form to document collective patterns across the panel, without turning the Observatory into a leaderboard.

Behind High Average Stability, Systems Do Not All Behave the Same Way

At first glance, the 12 observed systems may appear fairly similar: they often produce stable and coherent responses.

But a closer look shows that they do not all react in the same way.

It is a little like twelve cars driving on the same road: they are all moving forward, but some stay firmly in their lane, while others drift more — even if they may all appear calm from the outside.

In the NeoMundi baseline, observed stability ranges from 0.8005 to 0.8274 across profiles. Factual fragility signals range from 0.0018 to 0.0544. Semantic variation — meaning that the useful meaning of a response may change from one run to another — ranges from 0.0006 to 0.0723.

This leads to a simple conclusion: two AI systems can appear equally stable while showing different levels of variation and different levels of factual fragility.

The Barometer is therefore not designed to identify the “best” AI. It is designed to observe whether behaviours remain comparable, shift gradually, or begin to drift over time.

The Same Question May Not Produce the Same Answer

Across the 12 observed profiles, semantic variation is not distributed evenly.

For some profiles, it is almost absent. For others, it reaches 11.5%.

In practical terms, this means that, for certain systems, nearly one response in nine changes meaning enough from one execution to the next to be detected by the instrument — even though the question, test conditions and protocol remain comparable.

This is not necessarily an error. An AI system can answer differently while still remaining relevant.

But it is not neutral either.

In a professional setting — legal advice, healthcare, customer support, financial decision-making or autonomous agents — a changing answer may also alter a recommendation, a priority, an explanation or the level of confidence assigned to a decision.

The question is therefore not only: “Does the AI answer well?”
The real question is: “Does it answer predictably enough to be used and controlled?”

The Map of Observed Behaviours

Each point represents a de-identified profile.

The higher a point appears, the more its responses change meaning between two comparable executions. The further to the right it moves, the more factual fragility signals increase.

This map does not rank AI systems. It shows that systems which may appear similar can still display very different behavioural patterns.

This cartography is generated from aggregated public data using a reproducible Python script, available here: view the cartography generator.

Some AI systems change meaning enough, once in every nine answers, to be detected — even when asked the same question.

AI Behaviour Changes with the Type of Question Asked

The four questions tracked each week do not require the same type of effort:

• solving a short reasoning problem;
• explaining a scientific phenomenon;
• reflecting on the difference between stability and truth;
• reasoning in a zone of uncertainty.

For the first two, responses vary very little.
For the third, they remain relatively stable but show more factual fragility.
For the fourth, semantic variation reaches 10.54%: nearly one response in ten changes meaning enough to be detected.

This is intuitive: humans, too, would be more likely to give the same answer to a short calculation than to an open question about beliefs that may be wrong.

An AI system does not have one single level of reliability. Its behaviour also depends on the nature of the question it is asked.

Measurement Coverage: 27 Incomplete Cases

Out of 4,800 executions, 27 did not produce a complete set of measurements, representing 0.56%.

This is a small proportion, but it should not be hidden: an AI system may still provide an answer even when some of the signals needed to monitor it are unavailable.

In governance, an incomplete measurement is already part of the result.

What the Scores Show — and What They Do Not Prove

In this first campaign, several indicators — stability, validity and factual risk — are linked within the same calculation chain.

They help interpret a behaviour, but they do not constitute three independent pieces of evidence.

The Barometer does not turn scores into verdicts. It makes visible signals that must be interpreted together.