On the Limits of Selective AI Prediction: A Case Study in Clinical Decision Making
Sarah Jabbour, David Fouhey, Nikola Banovic, Stephanie D. Shepard, Ella Kazerooni, Michael W. Sjoding, Jenna Wiens · 2025
Read On the Limits of Selective AI Prediction: A Case Study in Clinical Decision Making by Sarah Jabbour, David Fouhey, Nikola Banovic, Stephanie D. Shepard, Ella Kazerooni, Michael W. Sjoding, Jenna Wiens (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07617.txt.
Introduction
Sarah Jabbour and six co-workers wrote a 2025 study on AI help in hospital care, under a head title about selective prediction. AI can aid doctors, but wrong AI calls can pull doctors into error. One fix is to hide AI calls that look unsure. The team tested if that fix works as hoped.
They ran a study with a large group of clinicians who diagnosed staged cases. Wrong AI calls lowered their scores next to no AI at all. Hiding unsure calls brought scores most of the way back. But when told the AI held back, clinicians missed more cases and treatments. The fix trades one kind of error for another.
Catalog record
| CREATOR | Sarah Jabbour, David Fouhey, Nikola Banovic, Stephanie D. Shepard, Ella Kazerooni, Michael W. Sjoding, Jenna Wiens |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-07617
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the Limits of Selective AI Prediction: A Case Study in Clinical Decision Making Sarah Jabbour, David Fouhey, Nikola Banovic, Stephanie D. Shepard, Ella Kazerooni, Michael W. Sjoding, Jenna Wiens Abstract AI has the potential to augment human decision making. However, even high-performing models can produce inaccurate predictions when deployed. These inaccuracies, combined with automation bias, where humans overrely on AI predictions, can result in worse decisions. Selective prediction, in which potentially unreliable model predictions are hidden from users, has been proposed as a solution. This approach assumes that when AI abstains and informs the user so, humans make decisions as they would without AI involvement. To test this assumption, we study the effects of selective prediction on human decisions in a clinical context. We conducted a user study of 259 clinicians tasked with diagnosing and treating hospitalized patients. We compared their baseline performance without any AI involvement to their AI-assisted accuracy with and without selective prediction. Our findings indicate that selective prediction mitigates the negative effects of inaccurate AI in terms of decision accuracy. Compared to no AI assistance, clinician accuracy declined when shown inaccurate AI predictions (66% [95% CI: 56%-75%] vs. 56% [95% CI: 46%-66%]), but recovered under selective prediction (64% [95% CI: 54%-73%]). However, while selective prediction nearly maintains overall accuracy, our results suggest that it alters patterns of mistakes: when informed the AI abstains, clinicians underdiagnose (18% increase in missed diagnoses) and undertreat (35% increase in missed treatments) compared to no AI input at all. Our findings underscore the importance of empirically validating assumptions about how humans engage with AI within human–AI systems. Preregistration — https://osf.io/m7avk Introduction AI models have been developed to assist human decision making in a variety of domains, such as child welfare ( De-Arteaga, Fogliato, and Chouldechova 2020 ) , wildlife conservation ( Beery, Morris, and Yang 2019 ) , and healthcare ( Nori et al. 2023 ; Gulshan et al. 2016 ; Adams et al. 2022 ; McKinney et al. 2020 ) . Yet, as AI-assisted decision-making becomes more prevalent, the risk of harm from incorrect AI predictions also grows. For example, automation bias, in which humans overrely on incorrect AI predictions, can propagate model errors through the human decision-maker. This is especially worrisome in high-stakes settings, such as healthcare, where it can result in downstream harm such as incorrect diagnoses ( Jabbour et al. 2023 ; Gaube et al. 2021 ) . Thus, researchers have focused on reducing overreliance and achieving appropriate reliance on AI: ensuring that decision-makers rely on AI when its predictions are correct, and ignore AI predictions otherwise ( Schemmer et al. 2023 ) . Figure 1: Left: In the multilabel setting, AI predicts several outcomes per patient, some of which are unreliable and may harm decisions (left). Right: Standard AI-assisted decision making shows all predictions, but risks automation bias. Instead, selective prediction shows only reliable AI predictions while hiding unreliable ones. To support appropriate reliance, researchers have explored interventions that nudge human decision-makers to make their own decisions when AI may be wrong. One common approach is selective prediction ( El-Yaniv and Wiener 2010 ) , where an AI may withhold its prediction on cases where it could harm the human’s decision. For example, in contrast to the standard workflow, in which the AI provides input for every decision (Figure 1 ‘Standard AI Workflow’), in selective prediction, some mechanism identifies predictions that are “unreliable” and will likely to lead to worse decisions (e.g., due to inaccuracy or uncertainty). In such cases, the system abstains from showing the prediction and asks the human to make a decision on their own (Figure 1 ‘AI + Selective Prediction’). When estimating human-AI team performance under selective prediction, it is often assumed that humans will respond exactly as they would without any AI involvement ( Nam et al. 2022 ; Farzaneh et al. 2023 ; Wang et al. 2023 ) . However, a key distinction exists between settings where the AI does not exist and those where the human is told the AI model has hidden its output . It is unclear if informing the human that the AI withheld its prediction alters human decision-making. Thus, in this work we empirically study the effects of selective prediction on human decisions in a real-world multilabel setting, using clinical diagnosis and treatment as a case study. This paper addresses the specific research question: Does selective prediction improve AI-assisted human decision making? To answer this, we conducted a study of 259 clinicians tasked with diagnosing the causes of hospitalized patients’ acute respiratory failure (shortness of breath). Acute respiratory failure (ARF) is a common condition that is often misdiagnosed and could benefit from AI-augmented clinical decision making ( Kempker et al. 2020 ; Zwaan et al. 2012 ; Jabbour et al. 2022 ) . Patients could have multiple concurrent causes, providing a useful test bed for multilabel selective prediction, where an AI model hides its prediction for a subset of conditions. We recruited clinicians who commonly care for these patients and randomized them to receive AI predictions with and without selective prediction. Contributions: This study is the first to empirically test selective AI prediction in the real-world context of clinical diagnostic and treatment decisions. We have two main findings. First, selective prediction mitigates the effects of inaccurate AI on human decisions regarding overall diagnostic and treatment accuracy. Second, selective prediction changes how clinicians navigate the tradeoff between false positives and false negatives: When AI abstains (and informs clinicians so), false positives decrease for patients who don’t need treatment, but false negatives increase for those who do . These findings underscore the need to empirically test AI deployment strategies and highlight that deployment decisions related to selective AI prediction must consider tradeoffs between the types of errors users are willing to accept. Background and Related Work Our work lies at the intersection of automation bias, selective prediction, and human-AI teaming. Mitigating Automation Bias. Prior work on mitigating automation bias has focused on designing AI interventions to help end users discern when to rely on the AI advice. In the field of explainable AI (XAI), explanations have been proposed to communicate the reasoning behind AI advice, to enable end users to ignore the AI when it is wrong and rely on it when it is right. However, the effectiveness of explanations has been mixed ( Lai and Tan 2019 ; Lai, Liu, and Tan 2020 ; Zhang, Liao, and Bellamy 2020 ; Jabbour et al. 2023 ; Bansal et al. 2021 ; Sivaraman et al. 2023 ) and can even mislead users into trusting flawed AI ( Lakkaraju and Bastani 2020 ) . Cognitive forcing functions–interventions that “slow down” thinking to make users focus during decision making ( Lambe et al. 2016 ) –have also been proposed to encourage users to engage more with the AI tools ( Buçinca, Malaya, and Gajos 2021 ; Gajos and Mamykina 2022 ) , although they may require a certain level of task expertise to reduce automation bias ( Banovic et al. 2023 ) . Researchers have also studied communicating uncertainty information about the AI prediction ( Kim et al. 2024 ; Dong and Hayes 2012 ; Gkatzia, Lemon, and Rieser