1. For AI systems, price, stability and factual accuracy do not always move together
We measured 12 major generative AI systems using the same protocol, in order to compare their observable stability, factual validity and relative cost per request.
Each point represents an observed profile. Horizontal axis: observable stability · Vertical axis: evaluated factual validity · Color: relative cost per request.
The results show that a more expensive system is not necessarily more stable or more accurate.
Conversely, a low-cost AI system can produce highly stable responses without ranking among the most reliable on factual validity.
Explore the data and methodology:
https://github.com/neomundi-io/…
Map of 12 systems by observable stability and evaluated factual validity; point color indicates relative cost (green low, orange intermediate, red high).
Observable stability →
Color = relative cost per request
Stability and validity barely predict one another (correlation ≈ 0.32), and cost overlaps with neither. Exploratory map: observed profiles, no public ranking of providers. It does not designate a “best model”; it shows different trade-offs.
NeoMundi · instrument view · June 2026
2. An AI system can be highly stable… and still be wrong
A system can respond very consistently when the same question is repeated, yet still fail to produce a factually correct answer.
In this campaign, generation stability is generally high, ranging from approximately 0.90 to 0.92. Yet factual validity varies significantly: from around 25% to 75%, depending on the observed profile and the judge used.
An AI system can remain consistent in the way it answers without being reliable on the substance.
One of the most stable candidates observed still achieves an average evaluated factual validity of approximately 61% on this targeted sub-panel.
Across the broad cohort, generation stability is tight (G-Score 0.90 to 0.92) while factual accuracy spreads from about 25% to 75%, both read on a full 0 to 100 scale. Stable does not mean accurate.
Generation stability (G-Score)
Factual accuracy (TruthfulQA)
Nominal price ≠ stability ≠ factual validity
One of the most stable candidates observed still achieves an average evaluated factual validity of approximately 61% on this targeted sub-panel, with a slight disagreement between the two automated judges.
3. The price of an AI system does not guarantee its reliability
Between the least expensive and the most expensive profiles observed in this campaign, the average cost per request varies by approximately ×300.
Yet the most expensive services are not systematically the most stable or the most factually accurate.
A massive price gap does not automatically translate into a reliability gap.
Average observed cost per request in this campaign, calculated from the consumption recorded by the pipeline and the pricing configured at the time of measurement. Provider pricing may evolve.
Average cost per request observed on the Wave 01 cohort · relative reading
4. Some errors remain silent
In our test, some systems produced relatively stable responses from one repetition to the next, while still achieving lower factual validity according to the judges.
In other words, a response can remain coherent, regular and apparently controlled without being fully reliable on the substance.
Stable response: G-Score 0.90 to 0.92, rare alerts. Yet about 47% of responses are judged non-compliant.
Stable response
0.90–0.92
G-Score · rare alerts
Yet
~47 %
of responses judged non-compliant
“Non-compliant” is not always a factual error (opinions, refusals) and calls for human validation. Broad TruthfulQA cohort · anonymized profiles · no provider names.
NeoMundi · AI Behavioral Cartography · June 2026
“Non-compliant” does not always mean “factually incorrect”: some responses may involve refusals, opinions or cases requiring human validation. TruthfulQA cohort · anonymized profiles · provider names withheld.
An error does not always come with an obvious warning signal.
When a system remains stable, users may have fewer reasons to question its response. Yet apparent stability is not enough to guarantee factual validity.
Measuring form does not replace verifying substance.
5. Who judges the judges?
Same response, two judges: nearly one verdict in five differs.
OpenAI Judge and Mistral Judge do not always return the same verdict.
Out of 9,087 responses evaluated by both judges, their conclusions diverge in 1,688 cases, representing 18.58% of the responses.
In this campaign, the difference is clearly directional: when the two judges disagree, Mistral Judge validates a response rejected by OpenAI Judge in 88.33% of cases.
Columns of the direction of disagreement between OpenAI Judge and Mistral Judge across 9,087 responses: 1,491 cases where Mistral accepts a response rejected by OpenAI, versus 197 reverse cases, out of 1,688 divergent verdicts (18.58%).
Disagreement rate
18.58%
Differing verdicts
1,688
Responses scored by both judges
9,087
The diagnosis also depends on the judge used. An automated verdict should therefore not be presented as an absolute truth, but as a signal measured within a given protocol.
Evaluating AI systems also means measuring the uncertainty of the evaluators.
The cartography is only beginning
This first edition is a starting point.
Each month, NeoMundi will publish new measurements and complementary analyses to better understand the real behavior of AI systems: what remains stable, what varies, what is costly, and what requires closer verification.
Future extractions may explore:
– informational density and the waste generated by responses;
– the variations observed when the same question is repeated at scale;
– disagreements between automated evaluators, with the gradual integration of additional judges;
– the emergence of behavioral profiles across use cases, providers and observed regimes.
These analyses will be produced by the NeoMundi team and progressively enriched by independent contributors.
Explore the data
The data, methodology and upcoming publications are publicly available:
Open data and methodology:
https://github.com/neomundi-io/ai-behavior-cartography
Test the measurement instrument:
https://controltower.neomundi.io
Contribute
NeoMundi is committed to an open science approach. Researchers, engineers, organizations and independent contributors can propose new questions, reproduce the measurements or suggest new lines of investigation.
We do not proclaim. We measure.