When Can We Trust Deep Neural Networks? Towards Reliable Industrial Deployment with an Interpretability Guide

Hang-Cheng Dong, Yuhao Jiang, Yibo Jiao, Lu Zou, Kai Zheng, Bingguo Liu, Dong Ye, Guodong Liu · 2026

Read When Can We Trust Deep Neural Networks? Towards Reliable Industrial Deployment with an Interpretability Guide by Hang-Cheng Dong, Yuhao Jiang, Yibo Jiao, Lu Zou, Kai Zheng, Bingguo Liu, Dong Ye, Guodong Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-19206.txt.

Introduction

Dong and seven colleagues propose a post-hoc flag for false negatives in defect networks. Trusting trained outputs is often unavoidable; one missed error can prove fatal. Their core idea contrasts class-specific and class-agnostic heatmaps, using their overlap gap as a score. Adversarial enhancement widens disparity. On two industrial benchmarks, it spots false negatives, reaching 100 percent recall with enhancement, at a cost to true negatives. It backs a data, model, explanation, output paradigm instead of the usual systems.

Catalog record

CREATORHang-Cheng Dong, Yuhao Jiang, Yibo Jiao, Lu Zou, Kai Zheng, Bingguo Liu, Dong Ye, Guodong Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24713
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.19206

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Can We Trust Deep Neural Networks? Towards Reliable Industrial Deployment with an Interpretability Guide Hang-Cheng Dong, Yuhao Jiang, Yibo Jiao, Lu Zou, Kai Zheng, Bingguo Liu, Dong Ye, Guodong Liu Abstract The deployment of AI systems in safety-critical domains, such as industrial defect inspection, autonomous driving, and medical diagnosis, is severely hampered by their lack of reliability. A single undetected erroneous prediction can lead to catastrophic outcomes. Unfortunately, there is often no alternative but to place trust in the outputs of a trained AI system, which operates without an internal safeguard to flag unreliable predictions, even in cases of high accuracy. We propose a post-hoc explanation-based indicator to detect false negatives in binary defect detection networks. To our knowledge, this is the first method to proactively identify potentially erroneous network outputs. Our core idea leverages the difference between class-specific discriminative heatmaps and class-agnostic ones. We compute the difference in their intersection over union (IoU) as a reliability score. An adversarial enhancement method is further introduced to amplify this disparity. Evaluations on two industrial defect detection benchmarks show our method effectively identifies false negatives. With adversarial enhancement, it achieves 100% recall, albeit with a trade-off for true negatives. Our work thus advocates for a new and trustworthy deployment paradigm: data-model-explanation-output, moving beyond conventional end-to-end systems to provide critical support for reliable AI in real-world applications. I Introduction Currently, deep learning methods have made significant advancements in both image [ 17 , 20 ] and language [ 19 ] domains. In particular, the remarkable success in the field of computer vision has enabled its application across various industrial scenarios, such as defect detection [ 1 , 18 ] , autonomous driving [ 21 ] , and medical diagnosis [ 6 ] . However, the inherent flaws of deep learning severely hinder its deployment in critical domains. The current deep learning paradigm lacks reliability, and a single error may lead to unacceptable catastrophic consequences. Even by increasing the volume of training data, impressive accuracy rates can be achieved, but this does not alter its black-box nature [ 10 ] . Without any intrinsic mechanism to proactively flag potential errors of the trained model, users can only blindly trust the model’s judgments. The deep learning paradigm can be summarized as a data-model-output pipeline. An intuitive approach is to assess the credibility of the output based on the magnitude of the model’s prediction. In addition, uncertainty estimation [ 11 ] methods have emerged that attempt to quantify model confidence using statistical sampling techniques [ 26 ] . Both types of methods essentially rely directly on the model’s output, with their core focus being on what the model does not know. Thus, they are often applied to out-of-distribution (OOD) detection [ 33 ] . Crucially, these methods often remain ineffective against predictions that are ”in-distribution yet confidently wrong,” and such failure modes represent a critical vulnerability in safety-critical applications. Can we determine when the network makes an error? Our core hypothesis is that the reliability of a prediction should be judged by the model’s internal reasoning process. Therefore, we turn our attention to the interpretability of deep learning. Post-hoc explanation methods such as Grad-CAM [ 24 ] can reveal the primary contributing regions that support the current model decision. Depending on their relevance to the output class, explanations can be categorized as discriminative or non-discriminative. This discrepancy inspires us to explore whether we can assess the reliability of the network’s decision by comparing different explanations for the same sample. Therefore, we re-examine post-hoc explanation methods and validate them in defect detection scenarios. To simplify the discussion, we refer to the two outputs of the defect detection problem as the background class and the defect class, corresponding to negative and positive samples, respectively. We find that different types of visual explanation methods produce discrepancies, which may stem from fundamental differences in the internal features they capture. We propose to quantify this discrepancy as the IoU difference between explanatory heatmaps, termed \Delta -IoU. As an indicator of model safety performance, \Delta -IoU can effectively identify false negative samples, but it can also lead to misclassification of true negative samples. Thus, we refer to samples with anomalous \Delta -IoU as suspicious samples. To enable \Delta -IoU to detect all false negative samples, we further propose an adversarial enhancement method. By performing adversarial attacks on the samples, subtle defect features are accentuated. To the best of our knowledge, this is the first proposed criterion for determining when a model’s output may be erroneous. The overall workflow demonstrates the irreplaceable role of interpretability research. Therefore, we argue that the classical data-model-output paradigm can be revised into a new paradigm: data-model-interpretation-output, which may lay the foundation for future AI applications in mission-critical domains. In summary, the contributions of this paper can be summarized as follows: • We propose a training-free method requiring no architectural modifications to identify suspicious samples, thereby preventing the potential severe harm caused by false negatives. • The proposed adversarial enhancement method achieves 100% recall in defect detection tasks, providing new insights for the application of deep learning methods in mission-critical domains. • We introduce interpretability as an integral component of deep learning model decision-making, integrating it into the decision-making process to form a new data-model-interpretation-output paradigm. Building upon this, we propose the novel task of suspicious sample detection, laying the foundation for applications in safety-critical domains. II Related work Uncertainty Estimation. Uncertainty estimation serves as a primary methodology for assessing the reliability of deep learning models [ 11 ] . The sources of uncertainty in network outputs can be categorized into aleatoric and epistemic types [ 16 , 15 ] . Techniques such as MC dropout [ 28 , 26 ] and deep ensembles typically quantify predictive confidence by measuring variations in model outputs or parameters. In essence, uncertainty estimation excels at identifying what a model does not know, making it particularly effective for tasks like out-of-distribution detection [ 33 ] . In contrast, this work focuses on instances where models are confidently wrong, which is a scenario we term suspicious samples. We define the task of identifying such potentially erroneous model outputs as suspicious sample detection (SSD). Confidence-based approaches like uncertainty estimation and our proposed suspicious sample detection are fundamentally distinct in both their objectives and methodologies. Explainable AI. Explainable AI (XAI) techniques aim to reveal the internal decision-making processes of deep learning models [ 10 ] . Based on how explanations are generated, these techniques can be categorized into built-in interpretable models [ 9 , 4 ] and post-hoc interpretation methods. Built-in interpretable models, such as B-CosNet [ 5 ] , achieve layer-wise feature explanations by eliminating bias terms. However, as most models lack this inherent property [ 23 ] , post-hoc interpretation methods are employed to uncover the internal decision logic of such models. A classic approach for feature interpretation is feature visualization [ 34 , 36 ] . For instance, gradient-based explanations [ 29 , 25 ] can reflect the contribution of input pixels to the final decision. Class activation mapping (CAM) [ 39 ] , on the other hand, enables analysis of the last convolutional features by assigning weights to channels based

Read the complete text