Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly ‘Know When They Do Not Know’
Shireen Kudukkil Manchingal Andrew Bradley Julian F. P. Kooij, Keivan Shariatmadar Neil Yorke-Smith Fabio Cuzzolin · 2025
Read Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly ‘Know When They Do Not Know’ by Shireen Kudukkil Manchingal Andrew Bradley Julian F. P. Kooij, Keivan Shariatmadar Neil Yorke-Smith Fabio Cuzzolin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-04950.txt.
Introduction
Manchingal and co-authors argue that AI systems will not be trustworthy until they can genuinely know when they do not know. Models ace their training territory yet stumble on unfamiliar or hostile inputs, and the usual ways of scoring their uncertainty have deep flaws. Their 2025 position paper calls for a shift to epistemic AI, built on second-order uncertainty, where a model tracks not just a prediction but the strength of the evidence behind it.
Such measures let a system separate what the data support from what is simply unknown, instead of hiding both in one confident number. The authors say this richer bookkeeping would make models sturdier in messy real settings, from shifting data to deliberate attacks. They frame current practice, fitting data ever harder, as the wrong race. The paper is a call to rebuild machine learning around admitted ignorance.
Catalog record
| CREATOR | Shireen Kudukkil Manchingal Andrew Bradley Julian F. P. Kooij, Keivan Shariatmadar Neil Yorke-Smith Fabio Cuzzolin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-04950
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Epistemic Artificial Intelligence is Essential for Machine Learning Models to Truly ‘Know When They Do Not Know’ Shireen Kudukkil Manchingal Andrew Bradley Julian F. P. Kooij, Keivan Shariatmadar Neil Yorke-Smith Fabio Cuzzolin Abstract Despite AI’s impressive achievements, including recent advances in generative and large language models, there remains a significant gap in the ability of AI systems to handle uncertainty and generalize beyond their training data. AI models consistently fail to make robust enough predictions when facing unfamiliar or adversarial data. Traditional machine learning approaches struggle to address this issue, due to an overemphasis on data fitting, while current uncertainty quantification approaches suffer from serious limitations. This position paper posits a paradigm shift towards epistemic artificial intelligence , emphasizing the need for models to learn from what they know while at the same time acknowledging their ignorance, using the mathematics of second-order uncertainty measures. This approach, which leverages the expressive power of such measures to efficiently manage uncertainty, offers an effective way to improve the resilience and robustness of AI systems, allowing them to better handle unpredictable real-world environments. 1 Introduction The success of artificial intelligence, especially deep learning [ 135 ] , is indisputable. AI systems now perform many tasks at or above human levels, with generative models advancing into creativity [ 192 ] , large language models (LLMs) [ 24 ] excelling in language manipulation, and significant advancements in multimodal AI [ 191 ] . However, this has also led to inflated expectations. For instance, autonomous vehicles have been touted as imminent breakthroughs for over a decade but technological challenges still remain a barrier to worldwide deployment [ 7 , 248 ] . While generative models like ChatGPT create remarkable outputs, there is an increasing acknowledgment of the need to reassess AI’s development path fundamentally [ 16 ] . Speculation about uncontrolled AI evolution and debates around artificial general intelligence (AGI) often overshadow pressing challenges AI must address today [ 202 ] . A significant limitation of current machine learning (ML) systems is their lack of robustness. Neural networks frequently make inaccurate and overconfident predictions when faced with uncertainties, such as out-of-distribution (OoD) samples, natural fluctuations, or adversarial disruptions [ 181 , 103 , 265 ] . These issues become safety-critical in autonomous vehicles due to models struggling to generalize across the diverse scenarios [ 86 , 22 ] the vehicle may encounter. While efforts in overfitting mitigation [ 221 , mu\vtop{\halign{#\cr\hbox{}\crcr}} ] and domain adaptation [ 89 ] are ongoing, these approaches are arguably insufficient to address the fundamental challenges of robustness in a meaningful manner [ 94 , 84 , 23 , 270 ] . There is a growing consensus that the accurate estimation of uncertainty [ 54 ] is vital to improve machine learning models' reliability [ 208 , 120 ] , with key applications to safety-critical areas such as autonomous driving [ 229 ] , medical diagnosis [ 133 ] , flood risk estimation [ 33 ] , and structural health monitoring [ 242 ] . To fully capture the uncertainty in a system or process, it is necessary to recognize two main sources: aleatoric (predictable, irreducible) and epistemic (unpredictable, reducible) uncertainty. The former arises from randomness in the data; a simple example of this is the coin-toss, where the data generating process has a stochastic component that cannot be reduced by any additional source of information [ 109 ] . The latter, instead, arises from a lack of knowledge about the system. For example, the odds of drawing the Ace of spades at random from a deck of cards might be assumed to be 1/52. However, this is based upon a prior assumption that this is a complete, standard deck. An `unknown' deck, however, may contain duplicates or missing cards, include jokers, or comprise multiple packs. Without this prior knowledge, the underlying model inevitably carries some uncertainty, which can be reduced with each subsequent observation. Hence, an awareness of the Socratic principle, to `know that you do not know', is of paramount importance. The main source of uncertainty in AI (but also in its science and engineering applications) is indeed the lack of a sufficient amount of data to train a model, in both quantity and quality ( i.e. , data fairly describing all regions of operation, including rare events). This uncertainty is epistemic in nature [ 55 ] , as it concerns the model itself, and can be reduced by collecting more data or information. While most scientists would agree that this is a profound problem, a defining issue for AI is how uncertainty should be managed, as existing uncertainty quantification (UQ) methods for AI have key limitations. Bayesian models are sensitive to prior mis-specification (with the risk of biasing the whole process) and incur heavy computational overhead [ 83 , 29 ] , while Bayesian Model Averaging (BMA) may dilute useful predictive information [ 105 , 96 ] . Ensemble methods are computationally demanding [ 115 , 102 ] . Conformal predictors primarily capture aleatoric uncertainty in a frequentist stance [ 15 ] . Evidential approaches violate asymptotic assumptions, struggle with out-of-distribution data [ 17 , 238 , 125 , 222 ] and exhibit high inference times (§ D ). This position paper advocates for a paradigm shift towards an Epistemic Artificial Intelligence emphasizing the importance of learning while acknowledging ignorance, using second-order uncertainty measures (§ A ) capable of overcoming those limitations thanks to their greater expressive power. Epistemic AI rests on the `paradoxical’ principle that one should first and foremost learn from (or be ready for) the data it cannot see. Prior to observing any data, the task at hand is thus completely unknown (albeit prior knowledge can be utilized to formalize the task and set a model space of solutions). The (limited) available evidence should only be used to temper our ignorance, to avoid `catastrophically forgetting’ how much we ignore about the problem. Epistemic AI is supported by both theoretical arguments and strong empirical evidence (Sec. 4 ). Firstly , the use of second-order uncertainty measures allows Epistemic AI to explicitly represent model ignorance and properly account for uncertainty due to lack of knowledge without biasing the learning process, unlike traditional approaches (Sec. 4.1 ). Secondly , evidence is recently mounting that Epistemic AI can predict uncertainty more accurately, at lower inference times (§ D ), and more broadly outperform other UQ methods in terms of accuracy, robustness and calibration (Sec. 4.2 ). As a result, Epistemic AI is capable of reducing the likelihood of AI systems being `surprised' by unexpected data or incapable to respond to unforeseen situations. This has enormous importance for mission-critical areas such as autonomous vehicles, or climate change and pandemic prediction, where long-term uncertainty is paramount as predictions concern the distant future and data is extremely scarce. Large language models learning `epistemically’ from data would be less likely to commit to false statements. Bias issues could be significantly mitigated, as epistemic models would not simply mimic the training data but account for possible future data. This paper presents arguments in support of Epistemic AI, discusses its potential and future challenges, while acknowledging alternative views. Paper structure. Sec. 2 shows how estimating uncertainty aids