VUS: Effective and Efficient Accuracy Measures for Time-Series Anomaly Detection
Paul Boniol, Ashwin K. Krishna, Marine Bruel, Qinghua Liu, Mingyi Huang, Themis Palpanas, Ruey S. Tsay, Aaron Elmore, Michael J. Franklin, John Paparrizos · 2025
Read VUS: Effective and Efficient Accuracy Measures for Time-Series Anomaly Detection by Paul Boniol, Ashwin K. Krishna, Marine Bruel, Qinghua Liu, Mingyi Huang, Themis Palpanas, Ruey S. Tsay, Aaron Elmore, Michael J. Franklin, John Paparrizos (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13318.txt.
Introduction
Paul Boniol and co-authors grade the graders of time-series anomaly detection in a 2025 paper. Spotting odd stretches in data feeds guards fraud, health, and machine systems. Yet scores borrowed from point-wise tasks can misjudge methods when faults span many time points.
A wide test of measures under noise, shifts, and varied fault shares shows threshold-free scores hold up best. The team extends those scores to span faults, then builds a family called VUS that sweeps settings with no knobs to set. Fair scoring, they argue, is a first step to honest claims of detection skill.
Catalog record
| CREATOR | Paul Boniol, Ashwin K. Krishna, Marine Bruel, Qinghua Liu, Mingyi Huang, Themis Palpanas, Ruey S. Tsay, Aaron Elmore, Michael J. Franklin, John Paparrizos |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 92 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13318
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VUS: Effective and Efficient Accuracy Measures for Time-Series Anomaly Detection Paul Boniol, Ashwin K. Krishna, Marine Bruel, Qinghua Liu, Mingyi Huang, Themis Palpanas, Ruey S. Tsay, Aaron Elmore, Michael J. Franklin, John Paparrizos Abstract Anomaly detection (AD) is a fundamental task for time-series analytics with important implications for the downstream performance of many applications. In contrast to other domains where AD mainly focuses on point-based anomalies (i.e., outliers in standalone observations), AD for time series is also concerned with range-based anomalies (i.e., outliers spanning multiple observations). Nevertheless, it is common to use traditional point-based information retrieval measures, such as Precision, Recall, and F-score, to assess the quality of methods by thresholding the anomaly score to mark each point as an anomaly or not. However, mapping discrete labels into continuous data introduces unavoidable shortcomings, complicating the evaluation of range-based anomalies. Notably, the choice of evaluation measure may significantly bias the experimental outcome. Despite over six decades of attention, there has never been a large-scale systematic quantitative and qualitative analysis of time-series AD evaluation measures. This paper extensively evaluates quality measures for time-series AD to assess their robustness under noise, misalignments, and different anomaly cardinality ratios. Our results indicate that measures producing quality values independently of a threshold (i.e., AUC-ROC and AUC-PR) are more suitable for time-series AD. Motivated by this observation, we first extend the AUC-based measures to account for range-based anomalies. Then, we introduce a new family of parameter-free and threshold-independent measures, Volume Under the Surface (VUS), to evaluate methods while varying parameters. We also introduce two optimized implementations for VUS that reduce significantly the execution time of the initial implementation. Our findings demonstrate that our four measures are significantly more robust in assessing the quality of time-series AD methods. ∎ Figure 1: Critical difference diagram computed with the Friedman test followed by a post-hoc Wilcoxon test (with \alpha=0.1 ) for the (a) F-score and (b) range-based F-score over 250 time series in KDD21 [ 28 ] . Bold lines indicate insignificant differences of connected methods. 1 Introduction Massive collections of time-varying measurements, commonly referred to as time series , have become a reality in virtually every scientific and industrial domain [ 5 , 39 , 44 , 43 , 47 , 19 , 6 , 42 , 40 ] . Notably, there is an increasingly pressing need for developing techniques for efficient and effective analysis of zettabytes of time series produced by millions of Internet-of-Things (IoT) devices [ 22 , 24 , 46 , 26 , 27 , 32 ] . IoT deployments empower diverse data science applications in environmental sciences, astrophysics, neuroscience, and engineering, among others [ 38 , 58 ] , and have revolutionized many industries, including automobile, healthcare, manufacturing, and utilities [ 37 ] . However, rare events, or imperfections and inherent complexities in the data generation and measurement pipelines, often introduce abnormalities that appear as anomalies in time-series databases, impacting the effectiveness of downstream tasks and analytics. Consequently, anomaly detection (AD) becomes a fundamental problem with broad applications sharing the same goal [ 7 , 51 , 56 ] : analyzing time series to identify observations that do not conform to some notion of expected behavior based on previously observed data. During the past decades, a multitude of AD methods have been proposed and compared [ 57 , 31 , 11 , 21 , 12 , 15 , 13 , 16 , 56 , 33 , 17 , 9 , 45 , 14 ] . Different from other domains that principally focus on point-based anomalies (i.e., outliers in standalone observations), AD for time series is also concerned with range-based anomalies (i.e., outliers spanning multiple observations). Unfortunately, it has become common practice to use traditional point-based information retrieval (IR) accuracy measures, such as Precision, Recall, and F-score, to quantify the effectiveness of different anomaly detectors. In addition, the previously mentioned IR evaluation measures suffer from a significant limitation: a threshold is necessary over the anomaly score produced by AD methods to mark each time-series point as an anomaly or not. The most common approach to set a threshold value is to use the average score plus three times the standard deviation of the anomaly score. However, this popular choice might not suit every AD method, use case, and domain, leading to significant variations in the quality values of the evaluation measures. Therefore, these IR measures are difficult to trust and complicate evaluating different AD methods on heterogeneous benchmarks. To eliminate the need to set a threshold, another standard measure for binary classification is used: the receiver operator characteristic (ROC) curve and the Area Under the Curve (AUC), which is the area below the ROC curve (AUC-ROC). The ROC curve is generated by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings (instead of only one threshold used in Precision, Recall, and F-score measures). Another variant, the Precision-Recall (PR) curve, represents the relation between Precision and Recall, and the Area under the PR curve (AUC-PR) is the area below PR [ 18 ] . Unfortunately, all previous measures, Precision, Recall, F-Score, AUC-ROC, and AUC-PR, are ideal for point-based anomalies but cannot adequately evaluate ubiquitous range-based contextual and collective anomalies [ 10 ] . Remarkably, the mapping of discrete labels into continuous data introduces unavoidable shortcomings (e.g., difficulty in marking precisely the range of the anomalies and handling misalignments between the human labels and the anomaly range produced by thresholding the anomaly score). To address these shortcomings, a range-based definition of Precision and Recall has been proposed by extending the traditional definitions [ 52 ] . Range-based Precision, Recall, and F-Score consider several factors: (i) whether a subsequence is detected or not; (ii) how many points in the subsequence are detected; (iii) which part of the subsequence is detected; and (iv) how many fragmented regions correspond to one real subsequence outlier. This definition is detailed and comprehensive; however, several parameters require tuning and, importantly, a threshold over the anomaly score is still required. A recent study [ 50 ] listed AD evaluation measures for time series, describing their advantages and shortcomings measured on synthetic time series. However, there has never been (to the best of our knowledge) a large-scale systematic quantitative and qualitative analysis of time-series AD evaluation measures on real time series. Notably, the choice of evaluation measure may significantly bias the experimental outcome. To understand the implications of choosing an appropriate measure, Figure 1 depicts the critical diagrams of the F-score and range-based F-score computed with the Friedman test followed by a Wilcoxon test [ 54 ] over several AD methods (see Section 5 for details) across the 250 time series of the KDD21 dataset [ 28 ] . Figure 1 demonstrates that not only the ranking is changing, but also some methods shift from insignificantly to significantly different from one measure to the other. In this paper, we extensively evaluate quality measures for time-series AD to assess their robustness under noise, misalignments, and different anomaly cardinality ratios. Specifically, our study includes 9 previously proposed quality measures, computed over the anomaly scores of 10 AD methods across 10 diverse datasets that contain 900 time series with marked anomalies. Our