Interpretable Failure Detection with Human-Level Concepts
Kien X. Nguyen, Tang Li, Xi Peng · 2025
Read Interpretable Failure Detection with Human-Level Concepts by Kien X. Nguyen, Tang Li, Xi Peng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05275.txt.
Introduction
A vision model that errs with full confidence can do real harm in a clinic or a car. Kien Nguyen, Tang Li, and Xi Peng spot such failures through human-level concepts in a 2025 paper. Instead of reading the model's final scores alone, they rank how strongly named concepts fire for each image.
That ranked pattern gives a finer sense of when a call is shaky. The method needs no extra parts or tuning tricks, the authors note. On standard image test sets, it cut false alarms by 3.7 percent on one and 9 percent on another. It also shows why a call looks wrong, in words people grasp.
Catalog record
| CREATOR | Kien X. Nguyen, Tang Li, Xi Peng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-05275
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Interpretable Failure Detection with Human-Level Concepts Kien X. Nguyen, Tang Li, Xi Peng Abstract Reliable failure detection holds paramount importance in safety-critical applications. Yet, neural networks are known to produce overconfident predictions for misclassified samples. As a result, it remains a problematic matter as existing confidence score functions rely on category-level signals, the logits, to detect failures. This research introduces an innovative strategy, leveraging human-level concepts for a dual purpose: to reliably detect when a model fails and to transparently interpret why . By integrating a nuanced array of signals for each category, our method enables a finer-grained assessment of the model’s confidence. We present a simple yet highly effective approach based on the ordinal ranking of concept activation to the input image. Without bells and whistles, our method significantly reduce the false positive rate across diverse real-world image classification benchmarks, specifically by 3.7 % 3.7\% on ImageNet and 9 % 9\% on EuroSAT . Department of Computer and Information Sciences University of Delaware Newark, DE, USA {kxnguyen, tangli, xipeng}@udel.edu Figure 1: Comparison between standard (MSP) and our approaches. MSP relies solely on class logits to predict failures, which is problematic in detecting overconfident but incorrect predictions. To tackle this problem, we propose to deconstruct each category into its associated human-level concepts for a finer-grained estimate of confidence. Introduction Vision-language models have demonstrated impressive capability across diverse visual recognition domains ( Radford et al. 2021 ; Jia et al. 2021 ; Singh et al. 2021 ; Li et al. 2022 ; Li et al. 2023 ) . However, when it comes to safe deployment in high-stake applications, it is of paramount importance for a model to be self-aware of its own shortcomings. For instance, in monitoring for natural disasters such as floods or wildfires, the AI system must signal for human intervention upon encountering scenarios where its confidence is low. Such self-awareness ensures that preemptive measures can be taken to mitigate disaster impacts on communities and ecosystems. Therefore, it is imperative not only to detect failures accurately but also to understand the reasons behind them. Traditional methods ( Hendrycks and Gimpel 2016 ; Granese et al. 2021 ; Zhu et al. 2023a ; Zhu et al. 2023b ; Liang, Li, and Srikant 2018 ) rely on category-level information to detect misclassifications, performing confidence estimation on the class logits. However, neural networks are known to produce overconfident predictions for misclassified samples due to factors like spurious correlations ( Arjovsky et al. 2019 ; Sagawa et al. 2019 ) , thus existing confidence scoring functions (CSFs) fall short in such cases. Besides, the model confidence depicted through category-level information impedes the ability for humans to interpret why it fails. To this end, we ask the following question: “What other sources of information can we leverage to enhance failure detection?” We present a novel perspective on detecting failures by leveraging human-level concepts, or visual attributes. With the flexibility to incorporate free-form language to VLMs ( i.e. CLIP), we can represent a category with a set of predefined concepts ( Menon and Vondrick 2023 ; Oikarinen et al. 2023 ; Li, Ma, and Peng 2024a ; Li, Ma, and Peng 2024b ) . Instead of only prompting the model “Do you recognize a camel?” , we collectively ask “Do you recognize humps on back?” , or “Do you recognize shaggy coat?” . The purpose is to measure the model’s confidence in the object’s detailed visual attributes in addition to the holistic category. We thus achieve a more accurate confidence estimate to detect failures more effectively (Fig. 1 ) Ideally, a VLM that can recognize a image of a camel should also recognize all the associated visual attributes, such as humps on back , shaggy coat , etc. Such visual attributes should yield higher confidence scores compared to those associated with the absent categories. Conversely, if the model shows high confidence in concepts from multiple unrelated categories at the same time, it could indicate a failure in its recognition process. Based on such intuition, we present a simple but effective approach using the O rdinal R anking of C oncept A ctivation (ORCA) to detect failures. Additionally, these human-understandable concepts allow users to understand the reasons behind such failures, thereby aiding them in refining the training process. We rigorously validate our method’s efficacy in detecting incorrect samples across both natural and remote sensing image benchmarks, which mirror the complexity in real-world scenarios. ORCA demonstrates a significant capability to mitigate the issue of overly confident misclassifications. In summary, our contributions are threefold: 1. We leverage human-level concepts to detect when and interpret why a model fails using vision-language models. 2. We present a simple but effective approach, called ORCA, to estimate more reliable confidence via the ordinal ranking of the concepts’ activation. 3. We empirically demonstrate that the concept-based methods enhance failure prediction performance across a wide range of classification benchmarks. Related Work Failure Detection. Failure detection, or misclassification detection, is a burgeoning area of research within the realm of artificial intelligence. Detecting when machine learning models produce incorrect or misleading predictions has significant implications for safety, reliability, and transparency in various domains. Existing research in this field falls into two main categories: (1) retraining or fine-tuning of neural networks ( Moon et al. 2020 ; Zhu et al. 2023a ; Zhu et al. 2023b ) , and (2) the design of novel confidence score functions ( Granese et al. 2021 ; Hendrycks and Gimpel 2016 ) . The former approach involves retraining or fine-tuning neural networks with specific objectives aimed at improving the model’s capability to recognize its own failures. Zhu et al. ( Zhu et al. 2023b ) employs a training objective that seeks flat minima to mitigate overconfident predictions. While these approaches have shown promise, they often require extensive computational resources and access to the entire model, which may not be feasible for large VLMs. Researchers have also turned their attention to the design of new CSFs ( Granese et al. 2021 ) . Despite these efforts, the most robust CSF remains the MSP ( Jaeger et al. 2023 ) . However, a downside of category-level CSFs is their inability to detect overconfident but incorrect predictions, which is problematic. In this work, we deconstruct category-level into concept-level signals to achieve a more nuanced estimate of the model’s confidence. A closely related sub-field is confidence calibration ( Minderer et al. 2021 ; LeVine et al. 2023 ; Mukhoti et al. 2020 ; Pereyra et al. 2017 ) , where the goal is to adjust a model’s predicted probabilities to ensure that they accurately reflect the true likelihood of those predictions being correct. However, Zhu et al. ( Zhu et al. 2023b ) has empirically shown that calibration methods frequently yield no benefits or even detrimentally affect failure prediction. Similarly, some works ( Jaeger et al. 2023 ; Bernhardt, Ribeiro, and Glocker 2022 ) also emphasizes the importance of confidence ranking over confidence calibration in failure detection. Some other related sub-fields are predictive uncertainty estimation ( Gal and Ghahramani 2016 ; Blundell et al. 2015 ; Lakshminarayanan, Pritzel, and Blundell 2017 ; Mukhoti et al. 2023 ) , out-of-distribution detection ( Zhu et al. 2023a