Beyond Accuracy: How AI Metacognitive Sensitivity improves AI-assisted Decision Making
ZhaoBin Li, Mark Steyvers · 2025
Read Beyond Accuracy: How AI Metacognitive Sensitivity improves AI-assisted Decision Making by ZhaoBin Li, Mark Steyvers (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22365.txt.
Introduction
An AI that knows when it is likely wrong can help a human more than a sharper but cocky one. ZhaoBin Li and Mark Steyvers formalize that claim in a 2025 paper. The trait at stake is confidence that sorts right calls from wrong ones.
Their framework weighs raw accuracy against that sorting skill jointly. Zones emerge where the less accurate but better-sorted AI wins. Team accuracy rises in those zones. A behavior test with real people confirms the gain. Scorecards for helper AI, they argue, should grade both traits. Tuning confidence may pay as much as tuning skill.
Catalog record
| CREATOR | ZhaoBin Li, Mark Steyvers |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22365
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Accuracy: How AI Metacognitive Sensitivity improves AI-assisted Decision Making ZhaoBin Li, Mark Steyvers Abstract In settings where human decision‐making relies on AI input, both the predictive accuracy of the AI system and the reliability of its confidence estimates influence decision quality. We highlight the role of AI metacognitive sensitivity—its ability to assign confidence scores that accurately distinguish correct from incorrect predictions—and introduce a theoretical framework for assessing the joint impact of AI’s predictive accuracy and metacognitive sensitivity in hybrid decision‐making settings. Our analysis identifies conditions under which an AI with lower predictive accuracy but higher metacognitive sensitivity can enhance the overall accuracy of human decision making. Finally, a behavioral experiment confirms that greater AI metacognitive sensitivity improves human decision performance. Together, these findings underscore the importance of evaluating AI assistance not only by accuracy but also by metacognitive sensitivity, and of optimizing both to achieve superior decision outcomes. keywords Human-AI collaboration; AI-Assisted Decision-Making; Metacognitive Sensitivity, Model Accuracy, Decision Analysis † † manuscriptno: DECA-0001-2024.00 † † runningauthor: Li and Steyvers † † runningtitle: How AI Metacognitive Sensitivity improves AI-assisted Decision Making † † authors: Department of Cognitive Sciences, University of California, Irvine, [email protected], [email protected] † † affiliation: † † affiliation: 1 Introduction Humans are increasingly relying on artificial intelligence (AI) for decision support in domains such as forecasting ( Benjamin et al. 2023 ) , clinical diagnosis, and judicial advisory services. The potential benefits of this collaboration stems from the complementary strengths and weaknesses of humans and AI. AI systems trained on large datasets can match or surpass human performance in narrow, specialized tasks, whereas humans excel at learning quickly from limited examples ( Lake et al. 2011 , Lee et al. 2015 ) , adapting to novel situations ( Franklin and Frank 2020 , Wu et al. 2024 ) , and juggling diverse objectives ( Goertzel 2014 ) . As a result, hybrid human-AI systems are expected to become increasingly common in everyday life. In AI-assisted decision-making, an artificial agent provides a recommendation that a human decision-maker may accept or override (see Figure 1 ). The challenge in this setup is to maximize expected utility, typically operationalized as the accuracy of the human’s final judgment. Achieving this goal requires appropriate reliance: humans should accept the AI’s advice when it is likely to be correct and reject it otherwise. However, humans usually lack direct knowledge of whether a particular AI recommendation is right or wrong. While overall performance metrics such as accuracy on benchmark datasets provide broad guidance, they do not inform users about the reliability of individual predictions. Instead, humans must rely on instance-specific signals—most notably confidence scores, explanations, and their own mental model of the AI’s behavior—to decide when to trust a recommendation ( Steyvers and Kumar 2023 , Lee et al. 2025 ) . Among these signals, confidence scores are especially valuable. Just as humans intuitively estimate the certainty of their own judgments ( Peters 2022 , Grimaldi et al. 2015 ) , many AI models output confidence estimates as a byproduct of their inference process. In classification settings, models such as logistic regression, naive Bayes, and neural networks with softmax outputs generate class probabilities directly. Ensemble methods like bagging or dropout approximate confidence via variability across model instances ( Breiman 1996 , Gal and Ghahramani 2015 ) . In large language models (LLMs), confidence estimation remains an open research area, with promising methods including token-wise likelihood analysis and sampling-based techniques ( Jiang et al. 2020 , Kadavath et al. 2022 ) . When these confidence estimates track correctness reliably, they can guide users toward more accurate decisions by signaling when the AI’s prediction is likely to be right or wrong. Figure 1 : Left: Illustration of the AI-assisted decision-making paradigm, in which a human integrates both the AI’s prediction and its associated confidence score to inform—and potentially improve—their final decision. Right: The AI’s metacognitive sensitivity measures the separation between its confidence distributions for correct and incorrect decisions. An AI with high metacognitive sensitivity is more likely to assign high confidence when it is correct and low confidence when it is wrong, compared to an AI with low metacognitive sensitivity. We show that an AI with high metacognitive sensitivity can assist humans in making better decisions. 1.1 Metacognitive Sensitivity versus Metacognitive Calibration The relationship between accuracy and confidence estimates can be characterized along two dimensions— metacognitive calibration and metacognitive sensitivity . Both play an important role in AI‐assisted decision‐making. Metacognitive calibration quantifies the correspondence between reported confidence and actual accuracy, measuring the degree to which the agent exhibits systematic overconfidence or underconfidence ( Fleming and Lau 2014 ) . Improved calibration in AI systems has been shown to enhance joint human–AI performance across a variety of tasks ( Ma et al. 2024 , Benz and Rodriguez 2023 , Zhang et al. 2020a ) , although some studies report exceptions under specific conditions ( Vodrahalli et al. 2022 ) . In contrast, metacognitive sensitivity measures how well an agent’s confidence discriminates between correct and incorrect predictions ( Fleming and Lau 2014 ) . In the forecasting literature, this is also referred to as discrimination ( Mandel and Barnes 2014 ) or resolution ( Murphy 1973 , Siegert 2017 ) . Recent work by Steyvers et al. 2025 demonstrates that LLMs with high metacognitive sensitivity—communicating their confidence effectively through verbal explanations—enable human collaborators to more accurately identify which AI responses are likely to be correct. Although related, calibration and sensitivity capture distinct aspects of metacognitive performance. Calibration reflects how closely confidence scores align with empirical accuracy and can often be improved post hoc via recalibration techniques such as Platt scaling ( Platt 1999 ) . Sensitivity, however, reflects the model’s intrinsic ability to rank correct predictions above incorrect ones and typically requires architectural or training modifications to enhance. 1.2 The Need for a Theoretical Framework Despite growing interest in how AI confidence interacts with human decision‐making, prior research has not explicitly isolated the role of AI metacognitive sensitivity nor examined its trade‐offs with base accuracy. In this research, we address this gap by proposing a theoretical model grounded in signal detection theory ( Maniscalco and Lau 2012 ) . This framework jointly models an AI’s accuracy and metacognitive sensitivity and analyzes their effect on the performance of a human decision‐maker assisted by the AI. In this framework, we derive an analytic expression for the human’s accuracy after receiving AI assistance as a function of both AI accuracy and metacognitive sensitivity. Analytical results under plausible assumptions reveal conditions in which an AI system with lower accuracy but higher metacognitive sensitivity may outperform a more accurate but less metacognitively sensitive AI in supporting human decision-making. We refer to this as an inversion scenario . We validate these theoretical predictions in a behavioral experiment in which human participants make perceptual judgments with AI assistance that vary systematically in both accuracy and sensitivity. The experimental results confirm the theoretical model, demonstrate the occurrence of inversion scenarios, and underscore the necessity of balancing both accuracy and metacognitive sensitivity to optimize AI‐assisted decision‐making. 2 Problem Setup We study a decision-making problem where a human decision-maker faces a multi-class classification task. For each instance, the human