SlowPoke: Understanding and Detecting On-Chip Fail-Slow Failures in Many-Core Systems
Junchi Wu, Xinfei Wan, Zhuoran Li, Yuyang Jin, Guangyu Sun, Yun Liang, Diyu Zhou, Youwei Zhuo · 2025
Read SlowPoke: Understanding and Detecting On-Chip Fail-Slow Failures in Many-Core Systems by Junchi Wu, Xinfei Wan, Zhuoran Li, Yuyang Jin, Guangyu Sun, Yun Liang, Diyu Zhou, Youwei Zhuo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24112.txt.
Introduction
Junchi Wu and seven co-authors present SLOTH, a tool to find slowdowns inside chips that run deep neural networks. Spatial accelerators are key for fast inference, but small fail slow faults are widespread and drag down performance. Finding them on the chip itself is hard because memory is tight and faults must be located within the chip layout.
SLOTH watches work at the level of single operations with little overhead and squeezes traces to fit in a few kilobytes. A ranking method that knows the chip topology then points to the likely root cause. Tests on many workloads show SLOTH cuts storage greatly while still detecting and locating the slowdowns well.
Catalog record
| CREATOR | Junchi Wu, Xinfei Wan, Zhuoran Li, Yuyang Jin, Guangyu Sun, Yun Liang, Diyu Zhou, Youwei Zhuo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24112
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SlowPoke: Understanding and Detecting On-Chip Fail-Slow Failures in Many-Core Systems Junchi Wu, Xinfei Wan, Zhuoran Li, Yuyang Jin, Guangyu Sun, Yun Liang, Diyu Zhou, Youwei Zhuo Abstract . Many-core architectures are essential for high-performance computing, but their performance is undermined by widespread fail-slow failures. Detecting such failures on-chip is challenging, as prior methods from distributed systems are unsuitable due to strict memory limits and their inability to track failures across the hardware topology. This paper introduces SlowPoke , a lightweight, hardware-aware framework for practical on-chip fail-slow detection. SlowPoke combines compiler-based instrumentation for low-overhead monitoring, on-the-fly trace compression to operate within kilobytes of memory, and a novel topology-aware ranking algorithm to pinpoint a failure’s root cause. We evaluate SlowPoke on a wide range of representative many-core workloads, and the results demonstrate that SlowPoke reduces the storage overhead of detection traces by an average of 115.9 × \times , while achieving an average fail-slow detection accuracy of 86.77% and a false positive rate (FPR) of 12.11%. More importantly, SlowPoke scales effectively across different many-core architectures, making it practical for large-scale deployments. 1 1. Introduction Many-core architectures, which integrate tens to thousands of processing cores on a single chip or via multi-chiplet integration, are becoming central to modern high-performance computing. Characterized by their massive parallelism and high aggregate throughput, these processors are designed to accelerate today’s most demanding workloads, including deep neural networks ( Zheng et al. 2021 ; Dhingra et al. 2025 ; Xu et al. 2025a ; Dhingra et al. 2024 ; Yin et al. 2023 ) , large-scale graph analytics ( Eyerman et al. 2018 ; Slota et al. 2015 ; Cong and Makarychev 2012 ) , and scientific simulations ( Merchant 2025 ; Tian et al. 2022 ; Ahmadzadeh and Sarbazi-Azad 2023 ) . As a result, many-core architectures are increasingly popular from edge platforms to supercomputers, often delivering significant gains in performance and energy efficiency compared to traditional multi-core CPUs ( Ltaief et al. 2023 ; Kundu et al. 2025 ; Wang et al. 2023 ; Wang and Martínez 2015 ) . (a) Fail-slow failures on different hardware components. (b) Normalized execution time of ResNet-50. Figure 1 . Fail-slow failure and its impact on overall performance. Execution times are measured on a 4 \times 4 core processor with a 10 \times performance degradation fail-slow of different components. However, the scale and complexity of many cores introduce a critical reliability challenge: the fail-slow failure ( Gunawi et al. 2018 ) . Unlike failures that result in system crashes ( Pillai et al. 2017 ; Alagappan et al. 2016 ) or incorrect outputs ( Clement et al. 2009 ) , a fail-slow failure allows a hardware component to remain functionally correct but operate at a significantly reduced speed. The causes are both complex and widespread, spanning multiple layers of the system. First, at the device level, physical degradation from effects like aging-induced transistor and interconnect wear can cause timing violations that reduce operational frequency ( Agarwal et al. 2007 ; Bhardwaj et al. 2012 ; Haghbayan et al. 2020 ; Wang et al. 2019a ) . Concurrently, thermal stress introduce dynamic system-wide slowdowns ( Zhou et al. 2019 ) . For example, under heavy workloads, extreme temperatures in DRAM can trigger protective states that cut performance by 24% to 52% ( Lin and Lin 2017 ; Shukla et al. 2019 ; Pandey et al. 2024 ) . Finally, bottlenecks emerge from shared resource contention and software bugs as cores compete for access to caches, memory, or the on-chip network ( Bjerregaard and Mahadevan 2006 ; Xiao et al. 2019 ) . The performance impact of these failures is especially severe in parallel applications where tasks are interdependent. In these workloads, progress is often dictated by the slowest-performing task—a phenomenon known as the straggler effect. A single slow core or link can stall numerous otherwise healthy components, leaving expensive parallel hardware idle ( Wu et al. 2024 ; Lin et al. 2025 ; Lu et al. 2022 ; Lu et al. 2023 ) . Our simulations quantify this cascading impact: as shown in Figure 1(b) , a single fail-slow event in a link, router, or core increases total execution time by 1.26 \times , 1.67 \times , and 2.48 \times , respectively. This demonstrates that fail-slow behavior can negate the benefits of massive parallelism, posing a critical barrier to building efficient and scalable many-core systems. Several approaches exist for detecting fail-slow failures in large-scale parallel and distributed systems. One common strategy is runtime tracing, which collects detailed event logs during execution ( Zhai et al. 2010 ; Khuvis et al. 2018 ) . Another is graph-based analysis, which constructs a dependency graph of the software tasks. By analyzing communication patterns and critical paths within this graph, these methods identify application-level stragglers that are holding up the overall computation ( Wang et al. 2022 ; Gan et al. 2024 ; Jin et al. 2020 ) . However, these strategies are not a good fit for on-chip detection due to two major challenges. First, runtime tracing is constrained by strict on-chip memory limits . For example, it takes 502\text{\,}\mathrm{KiB} and 2\text{\,}\mathrm{MiB} of storage to trace the communication and computation of a 16-core processor for just one second ( Figure 2 ), while a single core has typically less than 64\text{\,}\mathrm{KiB} of available SRAM. This makes logging impractical and and forces a difficult trade-off between monitoring overhead and accuracy. Second, graph-based analysis struggles with on-chip failure propagation . A slowdown in one hardware component quickly spreads to its neighbors, hiding the true source. Because existing graph-based methods analyze the software tasks and ignore the underlying hardware topology, they often misidentify the source, blaming a component that is merely slowed by a bottleneck elsewhere on the chip. Figure 2 . Challenges for detecting fail-slow failures in many-core processors: (a) On-chip, off-chip memory capacity versus computation and communication tracing log size for tracing DarkNet-19 for 1 second on a 4 \times 4 core processor. (b) Slowdown propagates across cores along communication dependencies. Therefore, an effective detection mechanism for many-core systems must be both lightweight to be efficient and hardware-aware to be accurate. A lightweight design must compress data on-the-fly and operate within tight memory limits, while a hardware-aware approach must analyze the chip’s hardware topology to trace a slowdown back to its true origin. In this paper, we present SlowPoke 1 1 1 The source code of SlowPoke along with failure datasets and traces are available at https://github.com/pku-lemonade/sloth , a fail-slow detection framework with a comprehensive workflow that spans data collection, storage, and analysis to meet these requirements. First, we propose an eBPF-like probe mechanism that enables lightweight and accurate performance data tracing without interfering with program execution. The probes are defined by a configurable tuple, allowing users to precisely specify insertion points. We further design a sketch-based data structure to compress and store the collected traces efficiently. To narrow down the failure candidates, we employ a group-based fail-core detection in combination with underdetermined linear system-based fail-link inference. To capture both software and hardware propagation of fail-slow failures, we reconstruct the communication dependency graph onto the hardware topology, and incorporate temporal information through hierarchical graph. Finally,