Accurate Evaluation of Quickest Changepoint Detectors via Non-parametric Survival Analysis

Taiki Miyagawa, Akinori F. Ebihara · 2026

Read Accurate Evaluation of Quickest Changepoint Detectors via Non-parametric Survival Analysis by Taiki Miyagawa, Akinori F. Ebihara (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18798.txt.

Introduction

Taiki Miyagawa and Akinori F. Ebihara give flexible estimators for quickest changepoint detection under short, uneven sequences. Average run length and detection delay guide theory and tests, yet real data rarely offer the long regular runs those criteria assume. Their KM-ARL and KM-ADD borrow survival analysis to model alarm chances under truncation. They prove bias bounds and stay unbiased unless extrapolation is needed, and tests on simulated and real data show sturdier estimates.

Catalog record

CREATORTaiki Miyagawa, Akinori F. Ebihara
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE118926
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18798

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Accurate Evaluation of Quickest Changepoint Detectors via Non-parametric Survival Analysis Taiki Miyagawa, Akinori F. Ebihara Abstract We propose non-parametric estimators for the average run length (ARL) and average detection delay (ADD) in quickest changepoint detection (QCD) under finite and irregular sequence lengths. Although ARL and ADD are widely used as optimality criteria in theoretical and simulation studies, their application to real-world datasets is hindered by limited and irregular sequence lengths. To address this issue, we propose non-parametric estimators for the ARL and ADD, termed KM-ARL and KM-ADD , by drawing an analogy between QCD and survival analysis to model detection probabilities under sequence truncation. We derive estimation bias bounds and prove that they are asymptotically unbiased unless extrapolation is required. Experiments on simulated and real-world datasets demonstrate their practical utility, enhancing robustness against limited and irregular sequence lengths, improving interpretability, and facilitating empirical, intuitive model selection. Our Python code is provided at https://github.com/TaikiMiyagawa/Kaplan-Meier-Average-Run-Length , offering ready-to-use implementations for practitioners. 1 Introduction Progress in machine learning depends not only on better models but on reliable performance measurement ( 37 ; 51 ; 52 ; 70 ) . Without principled estimation, improvements may be illusory and decisions may be incorrect. Therefore, reproducible and interpretable evaluation is a first-class concern in machine learning, not an afterthought. We study evaluation metrics for models in online quickest changepoint detection (QCD) with unknown pre- and post-change distributions, where their datasets with changepoint labels are available. QCD has been extensively studied theoretically, with many models proposed and their optimality proven ( 102 ) . It also has diverse real-world applications, including statistical process control ( 35 ) , industrial quality control ( 109 ) , epidemiology ( 42 ) , wireless sensor networks ( 31 ) , health monitoring ( 100 ) , radar target detection ( 113 ) , and seismic sensing ( 56 ) . The average run length (ARL) and the average detection delay (ADD) are central to the theoretical and simulation analysis of QCD models ( 101 ; 102 ) . The ARL is defined as the average time a detector takes to raise a false alarm, while the ADD refers to the average delay a detector takes to identify a changepoint after it has occurred. These metrics exhibit a tradeoff: reducing ADD typically shortens ARL (making the detector trigger-happy) and vice versa. Although the ARL and ADD are widely used as optimality criteria in theoretical and simulation studies, their application to real data is challenging because practical sequences are often limited and irregular in length. Fig. 1 (a) shows that naive ARL and ADD estimators yield substantial bias and variance, hindering reliable evaluation of QCD models. To address this issue, we draw an analogy between QCD and survival analysis ( 47 ) and propose non-parametric estimators for the ARL and ADD. We adapt the Kaplan-Meier estimator (KME) ( 43 ) to the QCD setting to model detection probabilities under sequence truncation. These estimators are non-parametric, requiring no assumptions on the underlying data distribution (e.g., exponential). We derive estimation bias bounds for these estimators, termed KM-ARL and KM-ADD , by decomposing the estimation bias into finite-sample bias and truncation bias . We then show that the finite-sample bias decays exponentially with increasing dataset size and that the truncation bias is smaller than that of conventional estimators. Building on these findings, we prove that the KM-ARL and KM-ADD are asymptotically unbiased unless extrapolation is required. Figure 1 : Evaluation of QCD models on real-world dataset (machine-labeled subset of WISDM Actitracker). (a) LB-ARL & LB-ADD. (b) KM-ARL & KM-ADD. Error bars represent the standard error of the mean. Our KM-ARL and KM-ADD are more robust to irregular lengths than the conventional estimators (LB-ARL and LB-ADD: see Sec. 3 ). The figures are shown at the same magnification scale for readability. The complete figure is provided in Fig. 14 in App. C.5 . Additional evaluation on a Gaussian process dataset is provided in App. C.3 . To demonstrate practical applicability, we conduct experiments on both simulated and real-world datasets with limited and irregular lengths. The results show that our estimators reduce estimation bias compared to baseline estimators and enhance robustness to limited and irregular sequence lengths, thereby improving interpretability and facilitating empirical, intuitive model selection (Fig. 1 (b)). Our code is provided at https://github.com/TaikiMiyagawa/Kaplan-Meier-Average-Run-Length , offering off-the-shelf, ready-to-use implementations in Python ( 108 ) for practitioners. Our contributions are threefold: (1) we propose KM-ARL and KM-ADD, non-parametric estimators for the ARL and ADD, enabling their evaluation on real-world datasets with limited and irregular-length sequences; (2) we derive estimation bias bounds for these estimators and prove that they are asymptotically unbiased unless extrapolation is required; and (3) we demonstrate their practical utility through experiments and provide ready-to-use Python implementations. 2 Related Work We highlight our contribution in the context of prior research on ARL and ADD estimation using survival analysis. To our knowledge, in QCD, no metric in its vanilla form explicitly accounts for irregular sequence lengths. 90 empirically estimate ARLs and ADDs on truncated sequences of fixed length, leveraging parametric survival analysis, assuming the survival function decays exponentially. 16 employ a semi-parametric survival analysis method, the Cox proportional hazards model ( 18 ) , which assumes an exponential response of the estimator to covariates. They also use the KME to empirically evaluate intrusion detectors under fixed length truncation; however, the ARL and ADD are not directly estimated, and no theoretical analysis is provided. 58 propose ad hoc estimators for the ARL and ADD on sequences of fixed length; however, their theoretical justification has yet to be established, as these estimators diverge when the sequence length \rightarrow\infty . In contrast, we build our estimators on non-parametric survival analysis, eliminating the exponential assumptions, derive bias bounds, and prove asymptotic unbiasedness for the proposed estimators. Moreover, our analysis includes irregular sequence lengths, thereby addressing more practical scenarios. We also provide a ready-to-use code of our estimators for practitioners. Supplementary related work is given in App. E . 3 Kaplan-Meier ARL & ADD We first introduce our notation. For comparison, we then introduce conventional estimators under random length truncation. Finally, we present our proposed estimators. Definitions. [n] denotes \{1,2,\ldots,n\} with n\in\mathbb{N} . We use P as a probability distribution. X^{(0,t)}=(X^{(0)},X^{(1)},\ldots,X^{(t)}) denote real-valued random variables representing a sequence of length t+1 ( t\in\mathbb{Z}_{\geq 0} ). Each frame X^{(s)} ( s\in\mathbb{Z}_{\geq 0} ) is sampled from the pre-change density g(X^{(s)}\mid X^{(0,s-1)}) when s<\nu and from the post-change density f(X^{(s)}\mid X^{(0,s-1)}) when s\geq\nu , where the changepoint \nu\in\mathbb{Z}_{\geq 0}\cup\{\infty\} is a random variable independent of the observations. \nu=\infty and \nu=0 indicate that all frames in the sequence are sampled from the pre-change and post-change densities, respectively. Let T\in\mathbb{Z}_{\geq 0} denote a random variable representing the sequence length, which is independent of the observations. Let \tau:X^{(0,T)}\mapsto\tau(X^{(0,T)})\in\mathbb{Z}_{\geq 0}\cup\{\infty\} be a changepoint detector, where \tau(X^{(0,T)}) is the detection point of sequence X^{(0,T)} . We set \tau(X^{(0,T)})=\infty when no change is detected within the input sequence of finite length. The ARL and ADD are defined as \displaystyle\,\mu_{\infty}:=\mathbb{E}[\tau\mid\nu=\infty] (1) and \displaystyle M_{\infty}:=\mathbb{E}[\Delta\tau\mid\Delta\tau\geq 0,\nu<\infty], (2) respectively, where \Delta\tau:=\tau-\nu . T=\infty is assumed here, and the expectation is also taken over \nu . We consider estimation of the ARL and

Read the complete text