The NeoMundi AI Observatory is our public research program dedicated to the continuous, transparent and reproducible evaluation of generative AI systems.
We regularly publish thermodynamic cartographies and in-depth observations, including the NeoMundi AI Barometer, which tracks de-identified model profiles over time across multiple dimensions: stability, factual validity, informational density, factual-risk signals, semantic variation and cost efficiency. These publications reveal behavioral trends and silent regime changes that traditional one-shot benchmarks cannot capture.
All our work is built on open methodologies and publicly available data. Generation scripts, scoring protocols and raw datasets are released on GitHub, allowing researchers, developers and the wider AI community to audit, reproduce and build upon our observations.
All AI Barometers Observation Level: 🟡 – This week, AI Barometer #6 explores two complementary dimensions. On the one hand, a form of volume variability: the same system can reach the same correct result while generating very different quantities of tokens. On the other hand, a return to calm after the peak observed the previous […]
NeoMundi Observatory ran a full runtime protocol on Kimi K3 (Moonshot): 1,200 valid executions – 120 situations repeated 10 times across 12 families (factuality, reasoning, governance, medical, legal, cybersecurity, business operations, and more). This is not a ranking.This is not a judgment.This is a runtime observation. The objective is to make visible what happens at
All AI Barometers AI Barometer #5: A behavioral regime change becomes observable Observation Level: 🟠 Orange This fifth edition of the AI Barometer highlights a significant behavioral shift compared to the previous period. The share of executions classified under the normal regime declined from 95.33% to 90.10%, while factual alerts increased from 1.31% to 5.88%.
⚠️ The first version of this Artificial Intelligence cartography combined two distinct protocols: a large-scale campaign conducted on twelve profiles (LLMs) submitted to 790 questions from TruthfulQA, and a complementary campaign based on twelve profiles, three categories of questions and 150 observations per category.This combination made it possible to observe multiple dimensions of system behavior
Companies no longer need to be told that artificial intelligence carries risks. They already know. They also know that a system can produce a convincing yet fragile answer, that a provider can modify a model or its routing without the change being immediately visible, and that an application can remain fully available from a technical
All AI Barometers When an AI responds consistently… but no longer says exactly the same thing This week, the Barometer highlights a particularly subtle form of instability: for the same question, the same system can generate multiple plausible, well-written, and seemingly logical responses… while varying the conclusion, nuance, or level of caution. None of the
By Frédéric Dumollard, former field inspector, then general agent and insurance broker. After years spent at the heart of underwriting corporate liability risks, he is now co-founder of NeoMundi and brings his expert perspective on the new insurance challenges posed by generative AI. DOI: 10.5281/zenodo.21222308 Why a former underwriter is taking up this subject I
All AI Barometers When a response sounds perfectly solid… but still warrants verification This week, the Barometer highlights a particularly insidious phenomenon: responses formulated with confidence, plausible at first glance, yet containing precise factual errors. Answers that many readers might accept without question. This is exactly what our measurement instrument detected on multiple occasions. Three
NeoMundi is pleased to announce its acceptance into the NVIDIA Inception program. An Already Operational Runtime Governance Infrastructure NeoMundi joins NVIDIA Inception to accelerate the development of ControlTower, our infrastructure for measurement, auditability, and real-time governance of AI systems. As organizations massively deploy AI in production, the key question is no longer whether a model
All AI Barometers One profile shows a clear shift in zone, while the overall picture remains calm This week, the overall average remains calm. However, one of the observed systems has changed its behavior and triggered our response verification control much more frequently compared to Barometer #1 last week. Indeed, the anonymized LLM with the
