Adaptive Confidence Regularization for Multimodal Failure Detection
Moru Liu, Hao Dong, Olga Fink, Mario Trapp · 2026
Read Adaptive Confidence Regularization for Multimodal Failure Detection by Moru Liu, Hao Dong, Olga Fink, Mario Trapp (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-02200.txt.
Introduction
Moru Liu, Hao Dong, Olga Fink, and Mario Trapp work on catching model failures in this 2026 paper. Models that mix image, text, and sound now serve in cars and clinics. When such a model fails, users need a warning, not just a wrong answer. The authors spot a clue: in most failures, the joint answer is less sure than one single-source answer.
Their method trains models to avoid that surety drop, which they call confidence loss. A loss term punishes any drop during training. They also swap features across sources to fake hard failure cases for practice. Across four data sets and three source types, the method gave steady, strong gains. It helps high-stakes systems flag and reject doubtful calls.
Catalog record
| CREATOR | Moru Liu, Hao Dong, Olga Fink, Mario Trapp |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63621 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.02200
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Adaptive Confidence Regularization for Multimodal Failure Detection Moru Liu, Hao Dong, Olga Fink, Mario Trapp Abstract The deployment of multimodal models in high-stakes domains, such as self-driving vehicles and medical diagnostics, demands not only strong predictive performance but also reliable mechanisms for detecting failures. In this work, we address the largely unexplored problem of failure detection in multimodal contexts. We propose Adaptive Confidence Regularization (ACR), a novel framework specifically designed to detect multimodal failures. Our approach is driven by a key observation: in most failure cases, the confidence of the multimodal prediction is significantly lower than that of at least one unimodal branch, a phenomenon we term confidence degradation . To mitigate this, we introduce an Adaptive Confidence Loss that penalizes such degradations during training. In addition, we propose Multimodal Feature Swapping , a novel outlier synthesis technique that generates challenging, failure-aware training examples. By training with these synthetic failures, ACR learns to more effectively recognize and reject uncertain predictions, thereby improving overall reliability. Extensive experiments across four datasets, three modalities, and multiple evaluation settings demonstrate that ACR achieves consistent and robust gains. The source code will be available at https://github.com/mona4399/ACR . 1 Introduction Multimodal models are increasingly adopted in safety-critical domains such as autonomous driving and medical diagnostics 14 ; 57 ; 20 . By integrating complementary cues from diverse modalities (e.g., video, audio), they often achieve superior robustness and generalization over unimodal approaches ( 18 ; 62 ) . However, even state-of-the-art models can be dangerously overconfident in their erroneous predictions ( 67 ) , posing serious risks in high-stakes applications. In such settings, detecting untrustworthy predictions is as crucial as achieving high overall accuracy. While prior work in uncertainty estimation ( 35 ) , calibration ( 22 ) , and out-of-distribution (OOD) detection ( 41 ) has aimed to mitigate overconfidence, these methods often fail to reliably flag individual predictions that should be rejected. Failure detection (FD) – also referred to as misclassification detection or selective classification – directly addresses this challenge by identifying unreliable predictions for potential rejection or human intervention, thereby reducing the risk of catastrophic failures ( 19 ) . While FD is well-established in unimodal settings, with methods spanning confidence-based scoring ( 21 ; 30 ) , outlier exposure ( 7 ; 73 ) , and confidence learning ( 10 ; 45 ) , its extension to multimodal systems remains largely unexplored. This gap is non-trivial, as unimodal approaches often fail to effectively leverage the complementary information across modalities or to handle failure modes unique to multimodal data, such as signal conflict and misalignment ( 50 ) . Furthermore, some works ( 16 ; 37 ) explore OOD detection with multiple modalities, but their settings fundamentally differ from those of FD. To illustrate the potential benefits of utilizing multiple modalities for FD, we present empirical results on the HMDB51 dataset ( 34 ) . All models in this analysis were trained solely with a standard cross-entropy loss. As shown in Figure 1 (left), a simple fusion of video and optical flow inputs substantially improves FD performance – measured by AURC, AUROC, and FPR95 – over unimodal baselines. This finding highlights the considerable potential of multimodal signals for improving FD . Concurrently, Figure 1 (right) reveals that sophisticated OOD detection methods like Energy ( 41 ) , Entropy ( 59 ) , and MaxLogit ( 25 ) are outperformed by a simple Maximum Softmax Probability (MSP) baseline ( 26 ) . Taken together, these findings demonstrate that merely adapting OOD techniques is insufficient and motivate the development of dedicated methods tailored for multimodal FD. Figure 1 : (Left) Multimodal models substantially enhance FD performance compared to unimodal models, without the need for complex designs. (Right) Advanced OOD detection methods underperform on FD tasks, while the simple MSP baseline surprisingly remains the most effective. In this work, we identify and systematically characterize the phenomenon of confidence degradation , a scenario where the confidence of fused multimodal predictions undesirably falls below that of individual unimodal predictions, particularly in misclassified instances. To address this, we propose Adaptive Confidence Regularization (ACR), the first dedicated framework for detecting failures (i.e., misclassifications) in multimodal systems. ACR comprises two key innovations: (1) an Adaptive Confidence Loss that explicitly penalizes confidence degradation during training, and (2) Multimodal Feature Swapping , a novel augmentation technique that synthesizes challenging, failure-aware training samples by swapping cross-modal embeddings. Training with the confidence penalty and failure-aware outliers improves the model’s ability to detect and reject uncertain samples, yielding gains in both accuracy and FD performance. Comprehensive experiments across five datasets and five modalities demonstrate that ACR sets a new state of the art, outperforming prior best methods by up to 9.58\% in AURC, 1.63\% in AUROC, and 15.45\% in FPR95. Further ablation studies under distribution shifts and multimodal OOD detection settings confirm the robustness and strong generalization of our approach. The primary contributions of this work are: • We highlight the importance of leveraging multimodal inputs for effective FD, and provide empirical evidence on the limitations of existing OOD detection approaches in this context. • We reveal and empirically validate the phenomenon of confidence degradation in multimodal models, showing its strong correlation with prediction failures. • We propose ACR, the first dedicated framework tailored to the complex task of multimodal FD. ACR integrates a novel Adaptive Confidence Loss, addressing the issue of confidence degradation , and introduces Multimodal Feature Swapping to further enhance confidence reliability. • We perform extensive evaluations across diverse datasets and modalities, demonstrating the robustness and effectiveness of ACR in a wide range of scenarios. 2 Methodology 2.1 Problem Setup Multimodal Failure Detection aims to detect misclassified samples using multiple modalities . We consider a training set \mathbb{D}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{n} drawn i.i.d. from the joint data distribution P_{\mathcal{X}\mathcal{Y}} , where \mathcal{X} is the input space and \mathcal{Y}=\{1,2,...,C\} is the label space. Each sample \mathbf{x}_{i} is composed of M modalities, denoted as \mathbf{x}_{i}=\{x_{i}^{k}\mid k=1,\cdots,M\} . Let f:\mathcal{X}\mapsto\mathbb{R}^{C} be a neural network trained on samples in P_{\mathcal{X}\mathcal{Y}} that predicts the label of each input sample. The f in multimodal failure detection comprises M feature extractors g_{k}(\cdot) and a classifier h(\cdot) . Each feature extractor g_{k}(\cdot) extracts an embedding \mathbf{E}^{k} for its corresponding modality k , and the classifier h(\cdot) takes the combined embeddings from all modalities as input and outputs a prediction probability \hat{p} : \begin{split}\hat{p}=\delta(f(\mathbf{x}))&=\delta(h([g_{1}(x^{1}),...,g_{M}(x^{M})]))\\ &=\delta(h([\mathbf{E}^{1},...,\mathbf{E}^{M}])),\end{split} (1) where \delta(\cdot) is the softmax function. We further include a classifier h_{k}(\cdot) for each modality k to get predictions from each modality separately, with the prediction probability from the k -th modality as \hat{p}^{k}=\delta(h_{k}(g_{k}(x^{k}))) . To safely deploy classifier f in real-world applications, it should not only be able to make accurate predictions but also distinguish and reject incorrect ones . Formally, let \kappa(\cdot) be a confidence-scoring function that quantifies the model’s confidence in its prediction. With a predefined threshold \tau\in\mathbb{R}^{+} , the misclassified samples can be detected based on a decision function G such that for a given input \mathbf{x} : G(\mathbf{x})=\left\{\begin{aligned} &\text{correct}~~~~~~~~~~\text{if}~~\kappa(\mathbf{x})\geq\tau,\\ &\text{misclassified}~~~~\text{otherwise}.\end{aligned}\right. (2) For example, we can easily use MSP ( 26 ) as the confidence-scoring function for a given input \mathbf{x} as \kappa(\mathbf{x})=\max_{y\in\mathcal{Y}}\hat{p} . Similarly, other confidence-scoring functions can be adapted from the OOD detection literature,