How to pick the best anomaly detector?
Marie Hein, Gregor Kasieczka, Michael Krämer, Louis Moureaux, Alexander Mück, David Shih · 2025
Read How to pick the best anomaly detector? by Marie Hein, Gregor Kasieczka, Michael Krämer, Louis Moureaux, Alexander Mück, David Shih (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-14832.txt.
Introduction
Marie Hein, Gregor Kasieczka, and four co-authors ask how physicists should pick the best anomaly detector for a given data set. Anomaly hunting can reveal new physics in regions no theory has targeted, but dozens of detectors compete and the winner depends on the data. Their answer is ARGOS, a data-driven score with a firm theoretical base.
ARGOS is tested on weakly supervised, classifier-based detectors, the style most used in particle physics searches. It beats older selection scores, including the standard cross-entropy loss, at picking the most sensitive model. The authors show it working for tuning settings, choosing network designs, and selecting input features, which makes it a practical tool for search teams.
Catalog record
| CREATOR | Marie Hein, Gregor Kasieczka, Michael Krämer, Louis Moureaux, Alexander Mück, David Shih |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-14832
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How to pick the best anomaly detector? Marie Hein, Gregor Kasieczka, Michael Krämer, Louis Moureaux, Alexander Mück, David Shih Abstract Anomaly detection has the potential to discover new physics in unexplored regions of the data. However, choosing the best anomaly detector for a given data set in a model-agnostic way is an important challenge which has hitherto largely been neglected. In this paper, we introduce the data-driven ARGOS metric, which has a sound theoretical foundation and is empirically shown to robustly select the most sensitive anomaly detection model given the data. Focusing on weakly-supervised, classifier-based anomaly detection methods, we show that the ARGOS metric outperforms other model selection metrics previously used in the literature, in particular the binary cross-entropy loss. We explore several realistic applications, including hyperparameter tuning as well as architecture and feature selection, and in all cases we demonstrate that ARGOS is robust to the noisy conditions of anomaly detection. I Introduction In recent years there has been an explosion of new methods for model-agnostic new physics searches at the LHC, leveraging powerful modern machine-learning (ML) anomaly detection methods such as autoencoders and weak supervision 34 ; 2 ; 10 ; 32 . These methods are able to find anomalies in the data with varying degrees of model independence: either signal-model-independence, in that they make no use of any specific new physics models in their training, and/or background-model-independence in that they make no use of Standard Model (SM) simulations in their training. As such, these new model-agnostic search strategies offer significant new opportunities for discovery in the vast, unexplored phase space of the LHC data, beyond what model-specific search strategies have been able to achieve to date. However, one important question has been left largely unaddressed to date: how does one pick the best anomaly detector from all the available options? In our case, we consider the “best” anomaly detector to be the most sensitive to unknown signals , as measured by the standard Significance Improvement Characteristic (SIC): \text{SIC}=\frac{\epsilon_{S}}{\sqrt{\epsilon_{B}}}, (1) where \epsilon_{S/B} is the signal/background efficiency after selecting the most anomalous events with the anomaly detector. The SIC is a multiplicative factor that quantifies the extent to which the anomaly detector enhances the significance of the new physics hiding in the data. Our goal is to select an anomaly detector that can achieve a high SIC sensitivity to any signal that is actually present in the given dataset. So far, when it comes to different approaches (e.g. autoencoders vs. weak supervision), the idea has been to just try them all, as seen in Ref. 14 . Within a given approach, there are myriad smaller but still vital choices to make: which model epoch to pick during training, which model architecture and hyperparameters to use, or even which features to use. Metrics traditionally used to guide these choices, such as the SIC and area under curve (AUC), are based on truth labels and require the use of benchmark signals. This is clearly not satisfactory for anomaly detection as the added signal dependence may, in the worst case, work against finding a signal actually present in the data. Consequently, none of the existing experimental anomaly detection analyses 6 ; 7 ; 8 ; 14 ; 1 ; 9 ; 15 ; 16 report systematic model optimization. Model architectures and training parameters are generally derived from the original publication introducing the corresponding method, with adjustments to network capacity to deal with the larger experimental datasets. In some cases 8 ; 14 ; 15 , parameters have been retuned using a small set of benchmark signals. In Refs. 14 ; 15 , anomaly scores have been selected from different classifier runs or epochs in certain approaches – CWoLa Hunting and TNT used the number of events selected in the signal region at a fixed background efficiency in the side bands as a metric; and CATHODE used the validation loss, following Ref. 28 . In this paper we introduce ARGOS (Above Random Gain Of SIC), a new fully data-driven metric for selecting the best anomaly detector. The key ingredient (besides the anomaly detector) that is required to calculate ARGOS is what we refer to as a background template (BT), which is a sample of events following the distribution of SM background in the signal region (SR). Then ARGOS is defined to be: \text{ARGOS}=\frac{\epsilon_{\text{SR}}}{\sqrt{\epsilon_{\text{BT}}}}-\sqrt{\epsilon_{\text{BT}}}, (2) where \epsilon_{\text{SR/BT}} is the efficiency to select SR/BT events for a given anomaly threshold. In contrast to SIC, which cannot be calculated for real (unlabeled) data, ARGOS is accessible for any anomaly detector if there is a corresponding background template in addition to the (unlabeled) data set. In Section II , we show that ARGOS has a sound theoretical foundation, being monotonic with the actual signal-to-background SIC for any unknown signal at a fixed background efficiency when the background template is perfect. More generally, ARGOS is proportional to SIC to a very good approximation in any meaningful setting. For concreteness, we demonstrate the utility of ARGOS within the context of weak supervision 37 , where a classifier is trained to distinguish two mixed-label data sets, for resonant anomaly detection. However, nothing in principle limits ARGOS to this context. Using the LHCO dataset 34 , introduced in Section III , we demonstrate how ARGOS can consistently select the best hyperparameters, architecture (e.g. neural networks vs. different boosted decision trees) and model epoch in Section IV . As an outlook, we also briefly discuss the potential for ARGOS to select the best features in a fully data-driven way before we conclude in Section V . Technical aspects of our experiments are discussed in several Appendices. II The ARGOS Metric In this Section, we show that ARGOS, as introduced in Sec. I , is the optimal data-driven metric for evaluating anomaly detectors when using an ideal BT. For this purpose, we rewrite the ARGOS metric as given in Eq. ( 2 ) in terms of the background and signal efficiencies using the probability density of the SR \displaystyle p_{\text{SR}}(x) \displaystyle=f_{S}\cdot p_{S}(x)+(1-f_{S})\cdot p_{B}(x)\,, (3) where f_{S} is the signal fraction in the signal region and p_{S/B}(x) denote the probability density for the signal/background events. For an ideal BT, we have \displaystyle p^{\text{ideal}}_{\text{BT}}(x) \displaystyle=p_{B}(x)\,, (4) which we try to approximate in real-world applications as discussed in Sec. III . Using these densities, which imply \epsilon_{\text{BT}}=\epsilon_{B} , we find ARGOS \displaystyle=\frac{f_{S}\cdot\epsilon_{S}+(1-f_{S})\cdot\epsilon_{B}}{\sqrt{\epsilon_{B}}}-\sqrt{\epsilon_{B}} (5) \displaystyle=f_{S}\cdot\left(\text{SIC}-\sqrt{\epsilon_{B}}\right), (6) Thus we see that ARGOS and SIC are closely related, at least in the idealized setting. In more detail, there are multiple ways one can view this result: • At fixed \epsilon_{B} , ARGOS is monotonic with SIC, since the signal fraction f_{S} is a constant. Then the best anomaly detector with respect to the ARGOS metric is also the best anomaly detector with respect to SIC. Additionally, ARGOS and the number of selected signal region events are also monotonic in this case and therefore also result in the same optimal decisions. The latter was used in Ref. 14 ; 15 to select epochs from multiple trainings of the CWoLa Hunting and TNT approaches. • For ARGOS, however, there is no need to fix the background efficiency a priori. For a good anomaly detector, \sqrt{\epsilon_{B}}\ll{\rm SIC} also holds, such that the