Formally Exploring Time-Series Anomaly Detection Evaluation Metrics
Dennis Wagner, Arjun Nair, Billy Joe Franks, Justus Arweiler, Aparna Muraleedharan, Indra Jungjohann, Fabian Hartung, Mayank C. Ahuja, Andriy Balinskyy, Saurabh Varshneya, Nabeel Hussain Syed, Mayank Nagda, Phillip Liznerski, Steffen Reithermann, Maja Rudolph, Sebastian Vollmer, Ralf Schulz, Torsten · 2025
Read Formally Exploring Time-Series Anomaly Detection Evaluation Metrics by Dennis Wagner, Arjun Nair, Billy Joe Franks, Justus Arweiler, Aparna Muraleedharan, Indra Jungjohann, Fabian Hartung, Mayank C. Ahuja, Andriy Balinskyy, Saurabh Varshneya, Nabeel Hussain Syed, Mayank Nagda, Phillip Liznerski, Steffen Reithermann, Maja Rudolph, Sebastian Vollmer, Ralf Schulz, Torsten (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17562.txt.
Introduction
Judging detectors that flag anomalies in time series turns out to be a swamp, because the scores used to grade them each capture only a sliver of what matters. Dennis Wagner, leading a team with twenty-six co-authors, puts that evaluation problem on formal ground. Missed anomalies can trigger disasters in chemical plants or power grids, so a misleading score is not a harmless academic flaw.
Wagner and his co-authors write down verifiable properties that any good metric should satisfy, then test thirty-seven widely used metrics against them. Most passed only a few properties, and none passed all, which explains years of contradictory rankings. To close the gap they propose a new metric that provably meets every property, plus a stricter variant for demanding uses.
Catalog record
| CREATOR | Dennis Wagner, Arjun Nair, Billy Joe Franks, Justus Arweiler, Aparna Muraleedharan, Indra Jungjohann, Fabian Hartung, Mayank C. Ahuja, Andriy Balinskyy, Saurabh Varshneya, Nabeel Hussain Syed, Mayank Nagda, Phillip Liznerski, Steffen Reithermann, Maja Rudolph, Sebastian Vollmer, Ralf Schulz, Torsten |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 190 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17562
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Formally Exploring Time-Series Anomaly Detection Evaluation Metrics Dennis Wagner, Arjun Nair, Billy Joe Franks, Justus Arweiler, Aparna Muraleedharan, Indra Jungjohann, Fabian Hartung, Mayank C. Ahuja, Andriy Balinskyy, Saurabh Varshneya, Nabeel Hussain Syed, Mayank Nagda, Phillip Liznerski, Steffen Reithermann, Maja Rudolph, Sebastian Vollmer, Ralf Schulz, Torsten Katz, Stephan Mandt, Michael Bortz, Heike Leitte, Daniel Neider, Jakob Burger, Fabian Jirasek, Hans Hasse, Sophie Fellenz, Marius Kloft Abstract Undetected anomalies in time series can trigger catastrophic failures in safety-critical systems, such as chemical plant explosions or power grid outages. Although many detection methods have been proposed, their performance remains unclear because current metrics capture only narrow aspects of the task and often yield misleading results. We address this issue by introducing verifiable properties that formalize essential requirements for evaluating time-series anomaly detection. These properties enable a theoretical framework that supports principled evaluations and reliable comparisons. Analyzing 37 widely used metrics, we show that most satisfy only a few properties, and none satisfy all, explaining persistent inconsistencies in prior results. To close this gap, we propose LARM, a flexible metric that provably satisfies all properties, and extend it to ALARM, an advanced variant meeting stricter requirements. 1 Introduction Many modern applications require the processing of data while it is generated. This includes monitoring patients, servers, or industrial production processes ( Darban et al. 2024 ) . We call data collected at regular intervals from one or more sensors time series . A fundamental task on time-series data is the detection of anomalies (TSAD) – rare yet significant, temporally bounded deviations from the norm ( Ruff et al. 2021 ) . TSAD plays a critical role in preventing catastrophic failures, increasing productivity, reducing losses, or ensuring safety. Therefore, the performance of different TSAD methods needs to be thoroughly evaluated. Figure 1: Illustration of a ground-truth anomaly (orange) and two prediction patterns (yellow). The upper prediction is timely and concise; the bottom prediction is delayed and oscillating, producing redundant alarms. Although both overlap equally with the true anomaly, they differ substantially in latency and redundancy—differences that many evaluation metrics ignore, often assigning them the same score. However, TSAD methods are often evaluated with inconsistent and incompatible metrics ( Lai et al. 2024 ; Wang et al. 2024 ; Kim et al. 2024 ) . Multiple works have shown that widely used metrics suffer from critical flaws, for example, significantly overestimating the performance of random models ( Doshi et al. 2022 ; Gim and Min 2023 ; Kim et al. 2022 ) , and several alternatives have been introduced to address such flaws ( Wagner et al. 2023 ; Bhattacharya et al. 2025 ; Si et al. 2024 ) . Yet, most metrics are tailored to specific applications and lack a formal foundation, making it difficult to compare or generalize their behavior across different settings ( Garg et al. 2021 ; Abdulaal et al. 2021 ; Gim and Min 2023 ; Sørbø and Ruocco 2024 ) . To enable trustworthy evaluations, metrics should behave in predictable ways that align with real-world needs. For example, a model that detects an anomaly is generally preferred over ones that fail to do so. Surprisingly, some commonly used metrics do not meet such requirements ( Section 5 , Table 1 ). his motivates a more systematic perspective: while definitive rankings of methods are generally difficult, meaningful comparisons often arise when methods differ in one key aspect, such as detection delay or alarm redundancy. From such pairwise comparisons, we derive general, verifiable properties that characterize the behavior of evaluation metrics. This characterization provides a useful tool for understanding the strengths and weaknesses of existing TSAD evaluation metrics while allowing for task-specific refinements. This paper makes the following contributions to the study of TSAD evaluation metrics: • A formal framework for evaluating TSAD metrics, introducing two sets of nine verifiable properties that capture relevant aspects of evaluation metrics such as detection accuracy, alarm redundancy, and timing. • A systematic theoretical study of 37 existing TSAD metrics, revealing that no metric satisfies all properties, thus explaining persistent inconsistencies in TSAD evaluations. • Two novel evaluation metrics —LARM and ALARM—that provably satisfy all proposed properties, allow flexible control over application-specific trade-offs, and yield intuitive, consistent model rankings in practice. 2 What makes a good metric? Metrics shape our perception of performance—and flawed metrics can distort it. This raises a fundamental question: what makes a good metric for TSAD? Choosing the right method for a TSAD application can be overwhelming. The number of available methods is huge and evaluations often vary considerably. Suppose we assemble a list of candidate methods and select an appropriate benchmark dataset on which we train our methods. Each produces predictions on the test set that we can compare with the ground truth. At this point, the central difficulty emerges: Which metric should we use? (a) (b) (c) (d) Figure 2: Challenging examples of prediction patterns (yellow) for different ground truths (orange) that common evaluation metrics struggle to order. Popular choices in the literature are point-wise Precision (the fraction of correctly predicted anomalous time steps among all predicted anomalous time steps) and point-wise Recall (the fraction of correctly identified anomalous time steps among all time steps labeled anomalous). By their point-wise nature, these metrics are unable to distinguish any patterns in the predictions such as depicted in Fig. 1 . Consider the example in Fig. 2 (a). Precision does not indicate a difference between the two methods, while Recall favors the second. In practice, however, we do not have access to the ground truth, and every alarm raised by the second method would require investigation, since they could be caused by distinct anomalies as illustrated in Fig. 2 (b), where neither Precision nor Recall indicate a difference in performance. A common adjustment treats any prediction within an anomaly window as correct if at least one time step in the same window is predicted correctly ( Xu et al. 2018 ) . With this adjustment, Recall favors the second method. However, this introduces further problems. Consider the example in Fig. 2 (c) and the adjusted predictions in Fig. 2 (d). Increasing the leftmost anomaly window makes the first method eventually appear better according to Recall, while Precision remains unchanged. However, the first prediction could be a single lucky guess, which becomes increasingly likely as the size of the anomaly window increases Doshi et al. 2022 ; Kim et al. 2022 ; Liu and Paparrizos 2024 . To illustrate the variation in rankings of different metrics, we compare the rankings of 16 predictions with 17 metrics in Fig. 3 . While the ranking on the left appropriately places perfect predictions first and effectively random predictions last, most metrics disagree considerably. Figure 3: The rankings of 16 synthetic methods assigned an individual color each from best (top) to worst (bottom) by the ALARM score ( Theorem 6.2 ) (left) compared to rankings of 15 other metrics (right), reveal that no two metrics agree on a ranking. These observations lead to our main hypothesis: Many TSAD evaluation metrics have flaws that are easily overlooked and counterintuitive. This highlights the need for a formal characterization of TSAD evaluation metrics and new evaluation setups that increase the transparency