Reliably detecting model failures in deployment without labels

Viet Nguyen 1,2,∗ , Changjian Shui 1,2, , Vijay Giri 3 , Siddarth Arya 1,2 , Amol Verma 1,4 , Fahad Razak 1,4 , Rahul G. Krishnan 1,2 · 2025

Read Reliably detecting model failures in deployment without labels by Viet Nguyen 1,2,∗ , Changjian Shui 1,2, , Vijay Giri 3 , Siddarth Arya 1,2 , Amol Verma 1,4 , Fahad Razak 1,4 , Rahul G. Krishnan 1,2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05047.txt.

Introduction

Viet Nguyen and co-authors tackled a hard problem for deployed machine learning in 2025: knowing when a model has started to fail. Data shift over time, and some shifts hurt a model while others do not. Without true labels, staff cannot simply check accuracy.

Their method, called D3M, watches for disagreement between predictive models. Under shifts that do not harm the main model, false alarms stay low. Under harmful shifts, the method gives sample bounds for a high rate of true detection. Tests on a standard benchmark and on a large internal medicine data set showed the approach worked well. The authors offer it as an alert tool for high-stakes systems.

Catalog record

CREATORViet Nguyen 1,2,∗ , Changjian Shui 1,2, , Vijay Giri 3 , Siddarth Arya 1,2 , Amol Verma 1,4 , Fahad Razak 1,4 , Rahul G. Krishnan 1,2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE97 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05047

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reliably detecting model failures in deployment without labels Viet Nguyen 1,2,∗ , Changjian Shui 1,2, , Vijay Giri 3 , Siddarth Arya 1,2 , Amol Verma 1,4 , Fahad Razak 1,4 , Rahul G. Krishnan 1,2 Abstract The distribution of data changes over time; models operating operating in dynamic environments need retraining. But knowing when to retrain, without access to labels, is an open challenge since some, but not all shifts degrade model performance. This paper formalizes and addresses the problem of post-deployment deterioration (PDD) monitoring. We propose D3M, a practical and efficient monitoring algorithm based on the disagreement of predictive models, achieving low false positive rates under non-deteriorating shifts and provides sample complexity bounds for high true positive rates under deteriorating shifts. Empirical results on both standard benchmark and a real-world large-scale internal medicine dataset demonstrate the effectiveness of the framework and highlight its viability as an alert mechanism for high-stakes machine learning pipelines. 1 Introduction Performance guarantees of conventional machine learning (ML) models hinge on the belief that the distribution of data with which these models train is identical to the distribution on which they are deployed ( rabanser2019failing ; recht2018cifar ; santurkar2020breeds ) . In many real-world scenarios such as healthcare, however, this assumption fails due to distribution shift during model deployment. Benchmarks such as WILDS ( koh2021wilds ) and WILD-Time ( yao2022wild ) have encouraged machine learning researchers to study and better understand how data shifts influence predictive systems. Yet, the number of tools at a practitioner’s disposal for building predictive models far exceeds those to monitor model failures. There is a need to create guardrails that self-detect and alert end-users to critical changes in the model when its performance drops below acceptable thresholds ( habib2021epic ; zadorozhny2022out ) . Post-deployment deterioration (PDD) monitoring presents a distinct set of systemic challenges stemming from considerations over the feasibility of deployment in real-world ML pipelines. Predominant is the scarcity of labels during deployment: for many downstream tasks such as in healthcare, labels are expensive to obtain ( razavian2015population ) or require human intervention ( liu2022deep ) . Due to deployed models predicting events temporally extended in the future ( boodhun2018risk ; zhang2019time ) , labels might even be unavailable. Another is the robustness of the monitoring system: it should flag critical changes in model deterioration early, using few samples, while remaining resilient to non-deteriorating changes to minimize unnecessary interruptions of service among other practical considerations. To address these challenges, we conceive a set of desiderata for any algorithm monitoring PDD, targeting their practicality and effectiveness as plug-ins to ML pipelines. To address the scarcity of labels, PDD monitoring algorithms should operate on unlabeled data from the test distribution to ascertain potential deterioration of the deployed model. Further, PDD monitoring algorithms should not depend on training data during deployment, as continuous (even indefinite) access to sensitive or personally identifiable training data might violate certain regulations protecting the privacy of data subjects ( muhlhoff2023predictive ) . An algorithm satisfying this desideratum is scalable, as it only audits a model’s input stream during monitoring with minimum data storage and regulatory concerns. Finally, PDD monitoring mechanisms should be robust to flagging non-deteriorating changes and effective in few-shot settings. When it comes to designing monitoring protocols that satisfy the above desiderata, recent related works only partially attend to individual desiderata. The literature on distribution shifts while achieving strong empirical performance on unlabeled deployment data ( liu2020learning ; zhao2022comparing ; zhang2024test ) , are not robust to false positives when the distribution shift is non-deteriorating. The model disagreement framework ( yu2019unsupervised ; chuang2020estimating ; jiang2021assessing ; ginsberg2023a ; rosenfeld2024almost ) emerges as a competitive setup for monitoring with downstream performance considerations via the tracking of disagreement statistics, while foregoing explicit distribution shift computations. However, shift-based and disagreement-based monitoring methods all depend on the presence of training data post-deployment , and do not provide any guarantees on robustness against false positives in the monitoring of non-deteriorating shifts. In this work, we answer all desiderata for PDD monitoring via the disagreement framework by proposing D isagreement- D riven D eterioration M onitoring (D3M) . Our contributions are as follows: Answering desiderata. D3M is a novel algorithm operating in the label-free deployment setting (1), requiring no training data during monitoring (2), and is provably robust in flagging deteriorating shifts as well as resilient to flagging non-deteriorating shifts (3). A comparison of the satisfaction of the PDD desiderata of our method with other related work in the literature is in Table 1 . Practical and scalable. D3M is model agnostic so long as the base model’s feature extractor can be optimized via gradient descent. This flexibility allows D3M to monitor various modalities of high-dimensional data. Unlike previous work, D3M avoids the retraining or finetuning of the base model via posterior sampling, crucial for the efficient monitoring of large models. Finally, owing to the decoupling of the algorithmic protocol into two distinct stages, D3M is efficiently scalable in the size of the training dataset , a critical consideration on the feasibility of its application onto current ML pipelines that is not enjoyed by standard baselines. Empirical validation. We showcase experimental results on various shift scenarios in the UCI Heart Disease dataset ( heart_disease_45 ) , CIFAR-10/10.1 ( recht2018cifar ) , Camelyon17 (WILDS) ( koh2021wilds ) , and an Internal Medicine (IM) dataset 1 1 1 For the moment, we have anonymized the name of the dataset and will be henceforth referred to as the Internal Medicine (IM) dataset. to demonstrate its effectiveness in monitoring models of various modalities. Our method effectively detects deployment-time deterioration with low false positive rates (FPR) when shifts are non deteriorating, and achieves competitive true positive rates (TPR) when shifts are deteriorating compared to standard baselines. In particular, we discuss how results on the internal medicine dataset suggest D3M to be well-suited for integration into real-world clinical monitoring pipelines. Provable algorithm. Under certain assumptions about the underlying distribution changes, D3M provably monitors model deterioration using finite samples when a deteriorating shift is present. In the presence of non-deteriorating shifts, D3M provably resists detection, thereby achieving low false positive rates. Table 1 : Comparisons between related work. Training data-free : whether post-deployment monitoring requires training data; Deteriorating : whether the method provably monitors the deteriorating shift; Non-deteriorating : whether the method is provably robust in the non-deteriorating shift; Disagreement : whether the method is based on the disagreement framework. Training data-free Deteriorating Non-deteriorating Disagreement Yu and Aizawa, 2019 yu2019unsupervised ✓ ✗ ✗ ✓ Liu et. al., 2020 liu2020learning ✗ ✗ ✗ ✗ Jiang et. al., 2021 jiang2021assessing ✗ ✗ ✗ ✓ Zhao et. al., 2022 zhao2022comparing ✗ ✗ ✗ ✗ Rosenfeld and Garg, 2023 rosenfeld2024almost ✗ ✓ ✗ ✓ Ginsberg et. al., 2023 ginsberg2023a ✗ ✓ ✗ ✓ D3M (ours) ✓ ✓ ✓ ✓ 2 Background and Algorithm 2.1 Problem setup Assume a base model f is supervisedly trained to classify inputs x\in\mathcal{X} into finite discrete classes \mathcal{Y}=\{1,\dots,C\} from training examples \mathcal{D}^{n}=\{x_{i},y_{i}\}_{i=1:n} where tuples (x_{i},y_{i})_{i=1:n}\sim\bm{P}^{n} for all i\in[n] . For a joint distribution \bm{P} over \mathcal{X}\times\mathcal{Y} , let \bm{P}_{x} denote its marginal distribution over \mathcal{X} . We are interested in designing a mechanism such that upon

Read the complete text