The NeoMundi AI Observatory is our public research program dedicated to the continuous, transparent and reproducible evaluation of generative AI systems.
We regularly publish thermodynamic cartographies and in-depth observations, including the NeoMundi AI Barometer, which tracks de-identified model profiles over time across multiple dimensions: stability, factual validity, informational density, factual-risk signals, semantic variation and cost efficiency. These publications reveal behavioral trends and silent regime changes that traditional one-shot benchmarks cannot capture.
All our work is built on open methodologies and publicly available data. Generation scripts, scoring protocols and raw datasets are released on GitHub, allowing researchers, developers and the wider AI community to audit, reproduce and build upon our observations.
If an AI system changes its behavior without changing its name, how would we notice? A single response is not enough. The same system can produce different answers to an identical question. Its behavior can also evolve over time, particularly when its model, settings, or infrastructure change. AI Weather™ makes some of these variations visible. […]
All AI Barometers Observation level: 🟠 Orange — insufficient coverage Of 4,800 executions launched across 12 systems, 3,981 were fully measured, representing 82.94% coverage, below the 90% standard publication threshold. Mean stability stands at 0.922, with an overall semantic variation rate of 1.61%. Two systems produced no usable observations despite multiple attempts, accounting for 800
The NeoMundi team is pleased to take part in an educational initiative designed to help students better understand AI Weather and the reliability issues associated with generative artificial intelligence systems. Follow the #AIweather on X This initiative is exclusively educational. Its purpose is to help students become better-informed users of artificial intelligence: to understand that
All AI Barometers Observation level: 🟡 Yellow Across 4,800 executions distributed among 12 systems, AI Barometer #12 remains overwhelmingly within the normal regime. Average stability remains virtually unchanged at approximately 0.923. The overall semantic variation rate stands at 1.41%, compared with 1.93% in Barometer #11. Coverage, however, decreased from 99.52% to 95.90%, with 197 incomplete
All AI Barometers Observation Level: 🟢 Green Across 4,800 executions distributed among 12 systems, AI Barometer #11 remains largely in the normal regime, with average stability virtually unchanged around 0.923. Global semantic variation reaches 1.93%, compared with 1.57% in Barometer #10. Factual alerts remain limited, and no generalized shift of the panel is observed. Reading:
James Moore’s analysis of AI Barometers #6 and #7 shows how a minimal global shift can mask concentrated changes across specific profiles, tasks and alert types. NeoMundi’s AI Barometers were designed to make the behaviour of AI systems observable over time. Once the measurement is produced, however, another question becomes essential: what can an apparently
All AI Barometers Observation Level: 🟢 Green Across 4,800 executions distributed among 12 systems, AI Barometer #10 remains largely in the normal regime, with average stability virtually unchanged at 0.9229. Global semantic variation decreases to 1.57%, compared with 2.09% in Barometer #9, while factual alerts remain limited. The observed movement remains mainly localized on a
Two systems with almost the same stability can differ by more than 30 points in factuality Protocol 1 – 12 systems × 790 TruthfulQA questions NeoMundi’s August monthly cartography once again compares twelve de-identified AI profiles across several complementary dimensions: their behavioral stability over 790 TruthfulQA questions, the factuality of their responses independently assessed by
All AI Barometers Observation Level: 🟢 Green Across 4,800 executions distributed among 12 systems, Barometer #9 remains largely in the normal regime (97.12%), with average stability virtually unchanged at 0.9228. Semantic variation, however, rises to 2.06%, compared with approximately 1.06% in the 8th Barometer, while factual alerts remain limited. The observed movement is mainly localized
All AI Barometers Observation Level: 🟢 Green This week, AI Barometer #8 confirms a return to global calm and moves to the green level. Two movements are particularly notable: This edition also presents a new example of a fragile factual claim embedded in a fluent and convincing response. This case illustrates the potential value of
