When Your Model Stops Working: Anytime-Valid Calibration Monitoring
Tristan Farran · 2026
Read When Your Model Stops Working: Anytime-Valid Calibration Monitoring by Tristan Farran (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13156.txt.
Introduction
Tristan Farran presents PITMonitor, an anytime-valid monitor for calibration drift in deployed probabilistic models. A fixed-sample test, run again and again on a stream, will false-alarm in time even for a stable model. Past tools also blur alarm time with the true change point. His monitor tracks shifts in probability integral transforms with a mixture e-process. That gives Type I error control over an endless horizon, plus a Bayesian change estimate. On a drift benchmark, detection matched top baselines.
Catalog record
| CREATOR | Tristan Farran |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35016 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13156
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Your Model Stops Working: Anytime-Valid Calibration Monitoring Tristan Farran Abstract Practitioners monitoring deployed probabilistic models face a fundamental trap: any fixed-sample test applied repeatedly over an unbounded stream will eventually raise a false alarm, even when the model remains perfectly stable. Existing methods typically lack formal error guarantees, conflate alarm time with changepoint location, and monitor indirect signals that do not fully characterize calibration. We present PITMonitor, an anytime-valid calibration-specific monitor that detects distributional shifts in probability integral transforms via a mixture e-process, providing Type I error control over an unbounded monitoring horizon as well as Bayesian changepoint estimation. On river’s FriedmanDrift benchmark, PITMonitor achieves detection rates competitive with the strongest baselines across all three scenarios, although detection delay is substantially longer under local drift. Code is available at https://github.com/tristan-farran/pitmon . keywords [class=MSC] Primary 62L10 ; secondary 62G10 keywords anytime-valid inference , calibration monitoring , e-values , e-processes , regime shifts , sequential hypothesis testing , changepoint detection , concept drift ?? University of Amsterdam 1 Introduction Probabilistic models deployed in production face a fundamental challenge: the world changes. Across domains, from medicine to finance, models encounter regime shifts and concept drift which can cause calibration to degrade drastically, with consequential effects downstream. Modern neural networks, for instance, can improve classification accuracy while simultaneously becoming less calibrated ( Guo et al. 2017 ) . Practitioners must therefore monitor their models continuously, seeking to answer two questions: has the world changed, and if so, when ? In practice, this monitoring often relies on ad-hoc procedures such as periodic recalibration schedules, rolling-window hypothesis tests, or threshold-based alerts on summary metrics. These approaches suffer from a fundamental statistical problem: they do not control the false alarm rate over continuous monitoring . A practitioner who checks calibration daily with a p<0.05 threshold will, over a year of monitoring, almost certainly observe spurious alarms even if calibration remains stable. Classical hypothesis tests assume a fixed sample size determined before seeing data; continuous monitoring violates this assumption. More principled alternatives are provided by online drift detectors, such as those implemented in the river library ( Montiel and Others 2021 ) . Classical detectors such as DDM and HDDM are lightweight and often effective at detecting abrupt changes, but rely on heuristic thresholds or fixed-sample statistical arguments that do not provide false alarm guarantees under continuous monitoring. ADWIN ( Bifet and Gavaldà 2007 ) improves on fixed-window methods by adapting its window size to bound the false alarm probability within a single window , but this per-window guarantee does not extend to the stream level: as the stream grows, the number of implicit tests accumulates without bound, inflating the false alarm rate. More fundamentally, these methods typically monitor generic signals such as error rates or mean changes in residuals rather than calibration-specific quantities, with broader notions of miscalibration such as overconfidence with unchanged accuracy potentially going undetected. We propose PITMonitor, an anytime-valid calibration monitor with five key properties: 1. Anytime-valid false alarm control: we prove that \mathbb{P}(\text{ever alarm}\mid H_{0})\leq\alpha for all time, without requiring a pre-specified monitoring horizon or stopping rule. 2. Calibration-specificity: we target the full calibration relationship between model and data, rather than indirect signals that may miss important aspects of miscalibration. 3. No baseline period required: unlike methods requiring a “clean” reference distribution or initialisation window, PITMonitor works from the first observation. 4. Practical efficiency: the method runs in O(1) per update of the mixture e-process, plus O(\log n) for rank maintenance via a SortedList ( Jenks 2014 ) , with O(n) memory. 5. Distribution-free: no assumptions are made about the pre- or post-change distributions. 2 Related Work Calibration Assessment Classical calibration metrics include expected calibration error ( Guo et al. 2017 ) , reliability diagrams ( DeGroot and Fienberg 1983 ) , and proper scoring rules ( Gneiting and Raftery 2007 ) . PITs have been used for static forecast evaluation in econometrics ( Diebold, Gunther and Tay 1998 ) and weather forecasting ( Gneiting and Katzfuss 2014 ) . These both provide static calibration assessment but do not address sequential monitoring with false alarm control. Distribution Shift Detection Methods for detecting distribution shift include two-sample tests ( Rabanser, Günnemann and Lipton 2019 ) and domain classifiers ( Lipton, Wang and Smola 2018 ) . These methods primarily address changes in the input or label distributions, while our work focuses on the output side: identifying when predicted probabilities no longer match outcome frequencies. Sequential Calibration Testing Arnold, Henzi and Ziegel 2023 proposed e-values for testing forecast calibration, focusing on whether PITs are \mathrm{Uniform}(0,1) . PITMonitor differs in two ways. First, we do not assume a specific PIT distribution and instead test whether the PIT distribution remains stable over time, alarming only on shifts in calibration rather than on miscalibration itself. Second, we locate when calibration changed, rather than only establishing whether the model is miscalibrated. E-values E-values support anytime-valid hypothesis testing under optional stopping ( Vovk and Wang 2021 ; Ramdas et al. 2023 ; Grünwald, de Heide and Koolen 2024 ; Shin, Ramdas and Rinaldo 2024 ) . PITMonitor builds on this framework, constructing a mixture e-process to detect shifts in the PIT distribution and inheriting its anytime-valid false-alarm guarantee (Theorem 1 ). 3 Background 3.1 Probability Integral Transforms A probabilistic model with predictive CDF \hat{F} for Y\sim P is perfectly calibrated if \hat{F}=P . Miscalibration , meaning divergence of predicted probabilities from outcome frequencies, can manifest as systematic bias, over- or under-confidence, misassigned tail risks, and more. The probability integral transform (PIT) U=\hat{F}(Y) unifies all of these into a single object computable from predictions and observations alone. Since U has distribution P\circ\hat{F}^{-1} , it encodes the calibration relationship between the model and data-generating process: every discrepancy between \hat{F} and P leaves a signature in the distribution of U . Perfect calibration corresponds to U\sim\mathrm{Uniform}(0,1) ( Dawid 1984 ) , but the shape of the PIT distribution is informative more broadly: a U-shaped density indicates underdispersion, a hump near the center overdispersion, and a triangular density systematic location bias ( Gneiting, Balabdaoui and Raftery 2007 ) . Thus, shifts in the PIT distribution indicate changes in calibration, making it a natural monitoring target. We therefore test: • \mathbf{H_{0}} : U_{1},U_{2},\ldots\overset{\mathrm{i.i.d.}}{\sim}F for some distribution F • \mathbf{H_{1}} : There exists a \tau>1 such that (U_{t})_{t<\tau}\overset{\mathrm{i.i.d.}}{\sim}F and (U_{t})_{t\geq\tau} is not i.i.d. from F As a result, PITMonitor is able to target any change in the calibration relationship, without alarming on static miscalibration. The null is broad, encompassing both perfectly calibrated models ( F=\mathrm{Uniform}(0,1) ) and miscalibrated yet stable ones ( F\neq\mathrm{Uniform}(0,1) ), while the alternative captures any change in calibration, whether abrupt or gradual, local or global. 3.2 Conformal P-values To detect distribution shifts sequentially, we employ conformal p-values from ranks ( Vovk, Gammerman and Shafer 2005 ) . Given observations U_{1},\ldots,U_{t} , define the rank of U_{t} as: R_{t}=\#\{s\leq t:U_{s}\leq U_{t}\} (1) Proposition 1 (Rank Uniformity Under H_{0} ) . 1 1 1 This result holds more generally under exchangeability of (U_{t}) ; we assume i.i.d. throughout for simplicity. If U_{1},\ldots,U_{t}\overset{i.i.d.}{\sim}F with F continuous, 2 2 2 If F is discretized or rounded, randomized tie-breaking can be used to preserve uniformity. then