July 2026 AI Cartography: Measuring the Behavior of 12 LLMs

⚠️ The first version of this Artificial Intelligence cartography combined two distinct protocols: a large-scale campaign conducted on twelve profiles (LLMs) submitted to 790 questions from TruthfulQA, and a complementary campaign based on twelve profiles, three categories of questions and 150 observations per category.
This combination made it possible to observe multiple dimensions of system behavior simultaneously, but it made the reading denser and could give the impression that the two protocols were measuring exactly the same thing.

📌 To improve clarity, readability and methodological rigor, we have therefore chosen to present these measurements separately from now on.

The cartography published here is based solely on:

• the 12 × 790 protocol, used to compare behavioral stability, response factuality and agreement between two independent AI judges,
• the 12 × 3 × 150 protocol, which will be the subject of a separate publication focused on stability under balanced and comparable conditions

This separation allows for a better understanding of what each campaign measures, avoids over-interpretation and makes the results more directly actionable.

Why a Stable AI Is Not Necessarily Reliable

Protocol 1 – 12 Systems × 790 TruthfulQA Questions

This cartography compares twelve de-identified AI profiles across several complementary dimensions: their behavioral stability on the 790 questions, the factuality of their responses evaluated separately by two AI judges (OpenAI and Mistral), as well as the level of agreement between these evaluators.

The map does not seek to designate a better system or establish a ranking of providers. It shows how profiles that appear very close on one indicator can differ significantly when another dimension is observed.

Each point represents a profile. Their position allows comparison of the regularity of their behavior with the accuracy of their responses. The two factuality evaluations then make it possible to observe whether the judges converge or whether certain responses remain more difficult to qualify.

What is immediately apparent is that the twelve systems are grouped within a relatively narrow range of stability, while their levels of factuality are much more dispersed. Some profiles therefore appear almost as stable as the best ones, while producing significantly more incorrect responses. For several of them, disagreement between the two judges also adds an area of uncertainty.

The map should thus be read as a simple demonstration: a single indicator is not enough to assess the reliability of an AI system. It is necessary to cross-reference stability, factuality, agreement between evaluation mechanisms and, beyond this snapshot, the evolution of behavior over time.

1. Stability alone creates an illusion of proximity

The twelve profiles display very tight stability levels, between 87.92% and 91.78%. At first glance, they therefore appear relatively comparable. But this behavioral proximity says nothing yet about the actual quality of the responses.

2. Factuality reveals much larger gaps

When observing the accuracy of responses, the gap becomes major: from 51.27% to 83.78% according to OpenAI as-a-judge. Two systems that are almost equally stable can therefore show more than thirty points of difference in factuality. Stability measures regularity, not truth.

3. Agreement between evaluators provides an additional signal

The two judges do not always qualify responses in the same way. The least factual profiles are also, in this campaign, those on which their disagreement is most marked. This divergence can signal ambiguous responses, difficult to verify or insufficiently robust to support a decision without human oversight.

4. A snapshot does not allow identification of drift

Even when crossing stability, factuality and judge agreement, the cartography remains a measurement taken at a given moment. It does not allow us to know whether a signal is occasional, whether it disappears or whether it persists. This snapshot must therefore be complemented by repeated measurements over time.

5. Governance begins with the cross-referencing of signals

Stability indicates whether behavior remains regular. Factuality indicates whether responses are correct. Evaluator agreement indicates whether their qualification is robust. Longitudinal monitoring indicates whether variations are becoming persistent. It is only by cross-referencing these dimensions that an organization can decide when to let the system operate, verify an output, trigger an investigation or reassess a use case.

---

Methodology: 12 anonymized systems (PROFILE codes), 790 TruthfulQA questions per system, double scoring by independent AI judges (OpenAI and Mistral). No provider identification is communicated publicly, in accordance with the Barometer’s editorial policy: measure, publish, never proclaim.

Protocol 2 – 12 Systems × 3 Iterations × 150 Questions

The first protocol observed the relationship between behavioral stability and factual reliability. This second protocol shifts the focus to another question: what do the signals produced during execution reveal, even before interpreting the quality of the final response?

The twelve de-identified profiles were submitted to three distinct series of 150 questions, i.e. 450 executions per profile and 5,400 executions in total. The objective here is not to evaluate the truth of the responses using external judgment, but to observe the conditions under which the systems operate: their stability, coherence, latency, measurement availability and any behavioral regime changes.

The map thus makes it possible to compare profiles beyond their final output. It shows which indicators really differentiate the systems, which remain constant, which profiles become more difficult to measure and where sufficiently strong variations appear to justify further investigation.

These observations address a directly operational challenge.

• An AI can continue to produce acceptable responses while becoming slower, less observable or more expensive to supervise.
• Conversely, a change detected during execution does not automatically constitute an anomaly: it must be placed back in context and confirmed over time.

Reading this map is therefore based on a simple principle: an isolated signal is not enough to draw a conclusion. It is the cross-referencing of stability, latency, measurement coverage, their differentiating power and the evolution of regimes that makes it possible to determine when a system can operate normally, when it needs closer monitoring and when revalidation becomes necessary.

1. An available metric is not necessarily a useful metric

Certain indicators remain identical across all profiles, notably coherence and normalized cost in this campaign. They are well measured, but do not allow systems to be differentiated or an action to be triggered.

Central idea: the value of an instrument does not depend on the number of metrics displayed, but on their ability to detect a situation that warrants intervention.

2. Two metrics with similar names can measure different realities

The internal factual signal in this protocol must not be confused with the external factuality measured in TruthfulQA by the OpenAI and Mistral judges.

Central idea: before interpreting or cross-referencing indicators, it is necessary to understand their definition, origin and the decision they actually allow one to make.

3. Measurability is a dimension of operational quality

PROFILE-DEA9C5 once again shows incomplete coverage, higher latency and several metrics that cannot be calculated. These observations do not prove a failure, but show that the system is more difficult to observe and supervise (see our 4th AI Barometer of the week).

Central idea: a system that is difficult to measure can generate invisible operational debt, even when it continues to produce responses.

4. Un changement de régime est un signal, pas un verdict

PROFILE-B912EA is the only profile to cross multiple behavioral regimes. This indicates that a change has occurred, but not whether it is positive, negative or simply linked to the corpus.

Central idea: a transition only becomes useful once it is placed back in context and monitored over time.

5. Metrology makes it possible to target human supervision

By cross-referencing stability, latency, coverage, metric availability and regime changes, the organization can concentrate its controls on genuinely atypical situations.

Central idea: measurement makes it possible to move beyond the alternative between blind trust and systematic human verification.

Data and methodology

The aggregated data, results tables and methodological documentation for this Cartography are available in NeoMundi’s public release:

GitHub: https://github.com/neomundi-io/ai-behavior-cartography/tree/main/releases/july2026-behavior-cartography
Public data for Protocol 1: 12 systems × 790 questions
Public data for Protocol 2: 12 systems × 150 questions × 3 iterations

The methodology describes, in particular, the execution protocol, de-identification rules, factuality evaluation process, coverage limitations and principles used to interpret the signals:

Methodology:
https://github.com/neomundi-io/ai-behavior-cartography/tree/main
The profiles remain de-identified, and no ranking of providers or models is published.