MERE: Hardware-Software Co-Design for Masking Cache Miss Latency in Embedded Processors

Dean You, Jieyu Jiang, Xiaoxuan Wang, Yushu Du, Zhihang Tan, Wenbo Xu, Hui Wang, Jiapeng Guan, Zhenyuan Wang, Ran Wei, Shuai Zhao, Zhe Jiang · 2025

Read MERE: Hardware-Software Co-Design for Masking Cache Miss Latency in Embedded Processors by Dean You, Jieyu Jiang, Xiaoxuan Wang, Yushu Du, Zhihang Tan, Wenbo Xu, Hui Wang, Jiapeng Guan, Zhenyuan Wang, Ran Wei, Shuai Zhao, Zhe Jiang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01582.txt.

Introduction

You leads a large team presenting MERE, a full system for runahead execution in embedded processors, in a 2025 paper. Runahead masks memory delays by pre-running code during long waits and prefetching data. It has mostly been used on large out-of-order cores.

Small in-order cores face tight area and energy limits plus severe cache contention. MERE rebuilds runahead with hardware and software working together, from chip and instruction set to operating system. An adaptive step eases cache contention. In tests, MERE reached most of the speed of a wider out-of-order core at a small added cost. The adaptive step gave a further clear gain.

Catalog record

CREATORDean You, Jieyu Jiang, Xiaoxuan Wang, Yushu Du, Zhihang Tan, Wenbo Xu, Hui Wang, Jiapeng Guan, Zhenyuan Wang, Ran Wei, Shuai Zhao, Zhe Jiang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-01582

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MERE: Hardware-Software Co-Design for Masking Cache Miss Latency in Embedded Processors Dean You, Jieyu Jiang, Xiaoxuan Wang, Yushu Du, Zhihang Tan, Wenbo Xu, Hui Wang, Jiapeng Guan, Zhenyuan Wang, Ran Wei, Shuai Zhao, Zhe Jiang Abstract . Runahead execution is a technique to mask memory latency caused by irregular memory accesses. By pre-executing the application code during occurrences of long-latency operations and prefetching anticipated cache-missed data into the cache hierarchy, runahead effectively masks memory latency for subsequent cache misses and achieves high prefetching accuracy; however, this technique has been limited to superscalar out-of-order and superscalar in-order cores. For implementation in scalar in-order cores, the challenges of area-/energy-constraint and severe cache contention remain. Here, we build the first full-stack system featuring runahead, MERE , from SoC and a dedicated ISA to the OS and programming model. Through this deployment, we show that enabling runahead in scalar in-order cores is possible, with minimal area and power overheads, while still achieving high performance. By re-constructing the sequential runahead employing a hardware/software co-design approach, the system can be implemented on a mature processor and SoC. Building on this, an adaptive runahead mechanism is proposed to mitigate the severe cache contention in scalar in-order cores. Combining this, we provide a comprehensive solution for embedded processors managing irregular workloads. Our evaluation demonstrates that the proposed MERE attains 93.5% of a 2-wide out-of-order core’s performance while constraining area and power overheads below 5%, with the adaptive runahead mechanism delivering an additional 20.1% performance gain through mitigating the severe cache contention issues. Journal: JACM Volume: 37 4 111 8 1. Introduction Driven by increasing demands for real-time performance and user privacy in modern computing applications, irregular workloads such as sparse machine learning and graph processing are not only executed on Out-of-Order (OoO) cores in data centers and desktop systems but also increasingly run on Scalar In-Order (Scalar-InO) cores within embedded devices for efficient local data processing ( Hua et al., 2023 ; STMicroelectronics, [n. d.] ; Jiang et al., 2021 ; Jiang et al., 2022 ; Fafoutis et al., 2018 ; Gharib et al., 2018 ; Shi et al., 2022 ; Sun et al., 2020 ; Chen et al., 2021 ; Cheng et al., 2021 ; Sun et al., 2021 ) . For instance, edge devices and Internet of Things (IoT) platforms leverage specialised Scalar-InO cores, e.g., ARM Cortex-M52 and Cortex-M55 ( ARM, 2023 ; ARM, 2020 ) , to perform sparse machine learning inference tasks. However, these workloads typically exhibit irregular memory access patterns (see Sec. 2 .1), resulting in frequent cache misses, which significantly prolong off-chip memory access times and degrade overall system performance ( Ainsworth and Jones, 2017 ; Ainsworth and Jones, 2018 ; Wang et al., 2025 ; Zhou et al., 2020 ; Bailey et al., 1991 ; Jasak, 2009 ; Izumo and Toivonen, 2004 ; Naithani et al., 2021 ) . Toward this, non-sequential memory access patterns brought by modern workloads and conventional hardware prefetchers (e.g., stream ( Srinath et al., 2007 ) and global-history buffers ( Srinivasan et al., 2004 ) ) have proved increasingly inadequate; Hence, considerable research has been devoted to runahead techniques ( Mutlu et al., 2003 ; Mutlu et al., 2005 ; Hashemi and Patt, 2015 ; Hashemi et al., 2016 ; Naithani et al., 2020 ; Naithani et al., 2021 ; Naithani et al., 2023 ; Naithani et al., 2024 ; Roelandts et al., 2024 ) . Runahead techniques pre-execute application code during occurrences of long-latency operations (i.e., runahead mode), where the processor frees pipeline resources and checkpoints the architectural register state, facilitating recovery after prefetching operations. Once the initial long-latency operation completes, the processor exits runahead mode, restores the checkpointed state, and resumes normal execution starting from the initial long-latency instruction. By prefetching the anticipated cache-missed data into the cache, runahead effectively masks memory latency for subsequent cache misses and achieves high prefetch accuracy even in irregular workloads (up to 95%) ( Hashemi et al., 2016 ) . While the runahead technique intuitively appears promising for masking the latency caused by irregular memory access in Scalar-InO processors, we found two fundamental incompatibilities when we tried to build it at the real RTL level: (i) Previous research demonstrated that runahead mechanisms add only about 0.5% \sim 2% area and 26.5% power overheads (primarily due to unbeneficial runahead durations) to a complex superscalar OoO core (e.g., ARM Cortex-A76 ( ARM, 2018 ) ), however, the modern Scalar-InO cores (e.g., ARM Cortex-M3 ( ARM, 2004a ) ) possess an order-of-magnitude smaller area/power-approximately 1% of the OoO cores. This means that, even modest overheads severely compromise the feasibility of directly integrating traditional runahead approaches into Scalar-InO cores. (ii) We observed that runahead techniques, which heavily rely on speculative execution, risk exacerbating cache pollution in Scalar-InO cores, due to the very limited cache capacity of the Scalar-InO cores. When a conflict prefetch (when multiple prefetch addresses map to the same cache set, subsequent prefetches displace earlier prefetched blocks in the Data cache (D-Cache) ) occurs, a future-required data block from the D-cache may be evicted, thereby inducing cache contention (see Fig. 3 ). Unlike the OoO cores, Scalar-InO cores cannot tolerate extensive speculative memory operations without risking severe cache pollution and subsequent performance degradation. To sum up, it is important but challenging to enable runahead in Scalar-InO cores, requiring a re-thinking of the methodology to manage irregular memory accesses effectively within resource-constrained Scalar-InO cores. Figure 1 . MERE reconstructs the architecture of sequential runahead, software and hardware, In software, miss number 4 is conflict with miss number 1, and miss number 5 is an L1 miss, miss number 1-4/6 are L2 misses. (SRH/ERH: Start/End Runahead; RCU: Runahead Control Unit; MC-CP: Multi-Cycle-CheckPoint; PMU: Prefetch Management Unit.) \name\ reconstructs the architecture of sequential runahead, software and hardware, In software, miss number 4 is conflict with miss number 1, and miss number 5 is L1 miss, miss number 1-4/6 is L2 miss. (SRH/ERH: Start/End Runahead; RCU: Runahead Control Unit; MC-CP: Multi-Cycle-CheckPoint; PMU: Prefetch Management Unit.) Contributions. In this paper, we show that it is feasible to build runahead into a real Scalar-Ino core with minimal area/power overheads while achieving high performance. To do so, we reconstruct the sequential runahead, employing a hardware/software co-design approach, trading off functionalities across system layers. This enables the runahead process to be precisely controlled, eliminates unbeneficial runahead duration, and allows conflict prefetch to be identified and skipped. We build a full-stack framework, Make Each Runahead Effective (MERE) , from the SoC and a dedicated Instruction Set Architecture (ISA) to the OS and programming model, providing a comprehensive solution for embedded processors managing irregular workloads. We deployed the proposed system on an AMD Alveo U280 FPGA and evaluated it using a range of metrics, including overall throughput, prefetching performance, and overheads. The experimental results indicate that implementing the MERE on Scalar-InO cores significantly improves the execution performance of irregular workloads. Our work achieves 93.5% of the performance of a 2-wide OoO core while limiting area and power overheads to under 5%, significantly outperforming superscalar OoO designs that incur double area and energy penalties (Fig. 3 ). Moreover, with an average performance improvement of 20.1% (Fig. 13 ), the adaptive runahead

Read the complete text