Detecting Anomalies in Systems for AI Using Hardware Telemetry
Ziji Chen, Steven W. D. Chien, Peng Qian, Noa Zilberman · 2025
Read Detecting Anomalies in Systems for AI Using Hardware Telemetry by Ziji Chen, Steven W. D. Chien, Peng Qian, Noa Zilberman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26008.txt.
Introduction
Ziji Chen, Steven W. D. Chien, Peng Qian, and Noa Zilberman propose finding faults in machine learning infrastructure from hardware signals alone. Cloud platforms hide user workloads from operators behind virtualization, which blocks system level optimization and hides misconfigurations. The authors argue that operators do not need workload knowledge to optimize the system, since low level hardware telemetry is fully in their view.
Their system, Reveal, feeds hardware signals into an unsupervised learning pipeline that flags anomalies. It was developed from analysis of over thirty popular models on varied hardware, so it adapts to new workloads. Reveal caught network and configuration problems in tests and sped up a large model run by about six percent after fixes.
Catalog record
| CREATOR | Ziji Chen, Steven W. D. Chien, Peng Qian, Noa Zilberman |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-26008
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Detecting Anomalies in Systems for AI Using Hardware Telemetry Ziji Chen, Steven W. D. Chien, Peng Qian, Noa Zilberman Abstract Modern machine learning (ML) has grown into a tightly coupled, full-stack ecosystem that combines hardware, software, network, and applications. Many users rely on cloud providers for elastic, isolated, and cost-efficient resources. Unfortunately, these platforms as a service use virtualization, which means operators have little insight into the users’ workloads. This hinders resource optimizations by the operator, which is essential to ensure cost efficiency and minimize execution time. In this paper, we argue that workload knowledge is unnecessary for system-level optimization. We propose Reveal, which takes a hardware-centric approach, relying only on hardware signals – fully accessible by operators. Using low-level signals collected from the system, Reveal detects anomalies through an unsupervised learning pipeline. The pipeline is developed by analyzing over 30 popular ML models on various hardware platforms, ensuring adaptability to emerging workloads and unknown deployment patterns. Using Reveal, we successfully identified both network and system configuration issues, accelerating the DeepSeek model by 5.97%. 1 Introduction Modern machine learning (ML) workloads, especially deep neural networks (DNN) training, have shaped recent compute infrastructure. These workloads rely heavily on compute-intensive operations which requires specialized accelerators. Large language models (LLMs) such as ChatGPT [ 1 ] , further push hardware limits [ 2 ] . The success of ML applications, such as in medical research [ 3 ] , has further driven demand for deploying ML workloads. However, these devices are expensive; a single GPU costs tens of thousands of dollars [ 4 ] , and together with the host system and maintenance, represents a considerable investment from cloud operators. This is demand for cloud-based AI resources is relentless, and is expected to grow 30% annually until 2030 [ 5 ] . Cloud operators provide users with environments to deploy ML applications using platform as a service (PaaS) [ 6 , 7 ] , hiding the underlying hardware through virtualization. In this way, operators can satisfy users with different resource requirements, provide elasticity, reliability, and isolation. While virtualization benefits cloud users, it leaves operators with limited workload knowledge. While important for privacy, security, and isolation, it is also problematic because modern ML application stacks have a tight hardware-software coupling. Individual components in the stack may be well optimized, but their integration is system- and workload-dependent. This is crucial, as small inefficiencies can cascade, leading to system-wide performance degradation and extended execution time. For example, distributed training relies heavily on collective communication, making the system prone to stragglers and network problems [ 8 ] . Even a minor power misconfiguration can cause up to 1.5\times slowdown [ 9 ] , reducing performance and increasing operational costs. Existing ML performance optimization works focus on a full-stack view [ 10 , 11 , 12 ] . It is difficult to diagnose performance anomalies without insights from the workload. Furthermore, what the user experiences inside a compute instance may differ from what happens on the hardware. To tackle the lack of high-level observability, we argue that having a full-stack view is in fact unnecessary to detect system-level anomalies. Instead, we take a hardware-centric approach by considering only hardware-level metrics, which operators have full access to. We propose Reveal, a hardware-centric profiling and anomaly detection framework. Reveal is lightweight and collects hardware metrics using existing tools such as perf [ 13 ] , widely available on Linux-based platforms. The framework is run on a bare-metal system by the operator, and sampled data is processed by an unsupervised anomaly detection pipeline. Reports are generated at intra- and inter-node levels, correlating anomalies across nodes. The Reveal anomaly detection pipeline was developed by studying over 30 popular ML models, running on systems with different hardware characteristics. We identified a subset of hardware metrics that reflect the ML workload’s behavior accurately. This design makes Reveal system- and workload- agnostic, supporting diverse hardware and both training and inference. Using Reveal, we successfully identified five hardware configuration issues when running the DeepSeek model on an HPC cluster, accelerating end-to-end time by 5.97%. In summary, we make the following contributions: 1. Introducing Reveal, a hardware-centric profiling and anomaly detection framework with high portability, deployability, and accurate analysis. 2. Identified a set of low-level performance metrics that are representative of ML workload’s behavior on hardware. We open-source all the collected datasets for future research. 3. Developed an unsupervised anomaly detection pipeline that detects performance problems running containerized ML workloads, successfully identifying system bottlenecks and accelerating DeepSeek by 5.97%. 2 Background and Motivation ML workloads dominate modern data centers [ 14 , 5 ] , driving deployment of specialized accelerators and interconnects which are programmed via vendor libraries. ML frameworks such as Torch [ 15 ] and TensorFlow [ 16 ] further abstract them from training and inference logic. This creates a full-stack ecosystem, spanning from hardware, network, system software, and applications. Performance anomaly analysis for ML workload is especially challenging because of the tight coupling between layers. A long GPU kernel launch time could be due to launching overhead rather than GPU saturation, and a low GPU utilization is meaningless without correlating the application timeline [ 11 ] . Similarly, a slow input preprocessing pipeline could be a result of disk misconfiguration or inefficient caching, rather than a CPU processing bottleneck [ 10 ] . Existing works of ML profiling (Table 1 ) stress the importance of having full-stack information [ 17 ] . The Hotline profiler [ 12 ] complements profiling provided by frameworks, such as Torch [ 18 ] , by merging and annotating their timeline with traces collected by other low-level runtimes, such as the CUDA profiling tool interface ( cupti ) and perf performance monitoring counters (PMCs). To improve coverage, RL-Scope [ 11 ] provides Python directives for developer to annotate critical paths in their code. Non-instrumentation tools such as Prometheus [ 19 , 20 ] , and AWS SageMaker [ 21 ] , do not modify applications. Instead, they sample operating system and hardware level statistics and perform anomaly analysis. Using these tools, ML developers can pinpoint where anomalies happen within a high-level workflow, specific functions, and attribute them to relevant hardware. Lack of high-level observability. Despite the availability of analytic tools, most are designed for ML developers, rather than cloud system operators. Due to the isolation, the operator lacks high-level observability, rendering tools that require application integration or running inside instances inapplicable. Due to elasticity, how and when users deploy the application becomes unpredictable. Cloud users are free to start, upscale, downscale, and stop their instances at any time. Workloads and scalability can also shift over time, for example, between training and inference dominance. This renders any one-size-fits-all configurations impractical. Our experiments (§ 6 ) revealed that misconfigured NIC interrupts could lead to GPU stalls, and unoptimized NVIDIA Collective Communications Library (NCCL) queue pair (QP) settings could lead to serialization. Both issues are detrimental to the performance of collective communication. Yet, without the perspective from the application, it is difficult to notice the issues. Lack of insight. Existing profiling tools primarily report observations, rather than analyze them, leaving anomaly detection and diagnosis to performance engineers (Table 3 ). Furthermore, many