Characterizing and Optimizing Realistic Workloads on a Commercial Compute-in-SRAM Device Conference: 58th IEEE/ACM International Symposium on Microarchitecture; October 18–22, 2025; Seoul, Republic of Korea 58th IEEE/ACM International Symposium on Microarchitecture (MICRO ’25), October 18–22, 2025,
Niansong Zhang, Wenbo Zhu, Courtney Golden, Dan Ilan, Hongzheng Chen, Christopher Batten, Zhiru Zhang · 2025
Read Characterizing and Optimizing Realistic Workloads on a Commercial Compute-in-SRAM Device Conference: 58th IEEE/ACM International Symposium on Microarchitecture; October 18–22, 2025; Seoul, Republic of Korea 58th IEEE/ACM International Symposium on Microarchitecture (MICRO ’25), October 18–22, 2025, by Niansong Zhang, Wenbo Zhu, Courtney Golden, Dan Ilan, Hongzheng Chen, Christopher Batten, Zhiru Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-05451.txt.
Introduction
Computing inside memory chips, rather than shuttling data to a processor, is benchmarked on real hardware by Niansong Zhang and six co-authors in a 2025 computer architecture paper. Compute-in-SRAM designs promise speed and energy gains, but most earlier evidence came from simulators or small prototypes.
The team measures a commercial device against ordinary processors and graphics cards under realistic workloads. An analytical framework exposes the main performance trade-offs, and three optimisations follow, covering how reductions are mapped, how transfers are merged, and how data are laid out. On retrieval tasks over huge corpora the system beat an optimised processor baseline several times over and matched a strong graphics card while using far less energy.
Catalog record
| CREATOR | Niansong Zhang, Wenbo Zhu, Courtney Golden, Dan Ilan, Hongzheng Chen, Christopher Batten, Zhiru Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-05451
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Characterizing and Optimizing Realistic Workloads on a Commercial Compute-in-SRAM Device Conference: 58th IEEE/ACM International Symposium on Microarchitecture; October 18–22, 2025; Seoul, Republic of Korea 58th IEEE/ACM International Symposium on Microarchitecture (MICRO ’25), October 18–22, 2025, Seoul, Republic of Korea DOI: 10.1145/3725843.3756132 ISBN: 979-8-4007-1573-0/2025/10 CCS: Hardware Emerging architectures CCS: General and reference Evaluation CCS: Computing methodologies Modeling methodologies Niansong Zhang, Wenbo Zhu, Courtney Golden, Dan Ilan, Hongzheng Chen, Christopher Batten, Zhiru Zhang Abstract . Compute-in-SRAM architectures offer a promising approach to achieving higher performance and energy efficiency across a range of data-intensive applications. However, prior evaluations have largely relied on simulators or small prototypes, limiting the understanding of their real-world potential. In this work, we present a comprehensive performance and energy characterization of a commercial compute-in-SRAM device, the GSI APU, under realistic workloads. We compare the GSI APU against established architectures, including CPUs and GPUs, to quantify its energy efficiency and performance potential. We introduce an analytical framework for general-purpose compute-in-SRAM devices that reveals fundamental optimization principles by modeling performance trade-offs, thereby guiding program optimizations. Exploiting the fine-grained parallelism of tightly integrated memory-compute architectures requires careful data management. We address this by proposing three optimizations: communication-aware reduction mapping, coalesced DMA, and broadcast-friendly data layouts. When applied to retrieval-augmented generation (RAG) over large corpora (10GB–200GB), these optimizations enable our compute-in-SRAM system to accelerate retrieval by 4.8 × \times –6.6 × \times over an optimized CPU baseline, improving end-to-end RAG latency by 1.1 × \times –1.8 × \times . The shared off-chip memory bandwidth is modeled using a simulated HBM, while all other components are measured on the real compute-in-SRAM device. Critically, this system matches the performance of an NVIDIA A6000 GPU for RAG while being significantly more energy-efficient (54.4 × \times -117.9 × \times reduction). These findings validate the viability of compute-in-SRAM for complex, real-world applications and provide guidance for advancing the technology. † † cc-license: by 1. Introduction Compute-in-memory (CIM) holds the promise of being a highly energy-efficient approach to accelerating data-intensive applications by reducing memory access overhead through the integration of compute units within or near memory arrays. Among CIM approaches, compute-in-SRAM stands out for its compatibility with standard CMOS technology and potential to achieve high memory bandwidth. The architectural and full-stack optimization of compute-in-SRAM systems continues to attract significant research interest. Recent works propose diverse designs: Compute Caches ( Aga et al. 2017 ) repurpose cache elements as vector units using bit-line SRAM, delivering 1.9 \times speedup and 2.4 \times energy savings over a 32-byte SIMD CPU; EVE ( Al-Hawaj et al. 2023 ) uses a bit-hybrid execution mechanism to accelerate vector operations by nearly 8 \times versus an out-of-order CPU; and CAPE ( Caminal et al. 2021 ) offers a programmable CMOS associative engine, averaging 14 \times speedup with peaks up to 254 \times over an area-equivalent CPU. Specialized accelerators such as iMTransformer ( Laguna et al. 2022 ) , TranCIM ( Tu et al. 2023 ) , PICMA ( Zhang et al. 2023 ) , and iMCAT ( Laguna et al. 2021 ) target deep neural networks (DNNs) and transformer models, highlighting the potential for domain-specific acceleration. Table 1 . Comparison of GSI APU ( Technology 2023 ; Gwennap 2020 ) , Intel Xeon 8280, NVIDIA A100, and Graphcore IPU. GSI APU Xeon 8280 NVIDIA A100 Graphcore Compute Cores 2 million \times 1 bit 28 \times 2 \times 512 bits 104 \times 4,096 bits 1,216 \times 64 bits Tech node 28 nm 14 nm 7 nm 7 nm Compute Speed 500 MHz 2.7 GHz 1.4 GHz 1.6 GHz Peak Compute 25 TOPS 10 TOPS 75 TOPS 16 TOPS On-Chip Memory 12MB L1 38.5MB L3 40MB L2 300MB L1 Mem. Bandwidth 26 TB/s 1 TB/s 7 TB/s 16 TB/s Power 60W TDP 205W TDP 400W TDP 150W TDP Figure 1 . General-purpose compute-in-SRAM system model – CAPE ( Caminal et al. 2021 ) , and GSI APU ( Technology 2023 ; Gwennap 2020 ) follow the same system abstraction, with different vector arithmetic unit and SRAM cell implementations. Despite promising results, these architectures are primarily evaluated through instruction modeling and simulation ( Aga et al. 2017 ; Zha and Li 2020 ; Laguna et al. 2022 ; Laguna et al. 2021 ; Zhang et al. 2023 ) or small-scale prototypes ( Tu et al. 2023 ) , limiting insights into their practical, real-world effectiveness. This gap underscores the need for performance characterization of commercial compute-in-SRAM devices under realistic workloads, bridging the divide between theoretical promise and practical feasibility. General-purpose compute-in-SRAM systems typically employ a SIMD vector processor abstraction ( Aga et al. 2017 ; Caminal et al. 2021 ; Zha and Li 2020 ) . As illustrated in Fig. 1 , a common compute-in-SRAM architecture—adopted by systems like CAPE—abstracts the computation-enabled SRAM array as a vector processing engine. The GSI APU ( Technology 2023 ; Gwennap 2020 ) aligns with this same abstraction, representing a commercial instance that provides a unique opportunity to evaluate the potential of compute-in-SRAM systems under realistic workloads and applications. The GSI APU integrates 2 million bit processors at 500 MHz, delivering up to 25 TOPs for 8-bit addition ( Technology 2023 ) . Table 1 compares its compute capacity, memory bandwidth, and power efficiency against CPUs, GPUs, and ASIC accelerators, showing strong potential for data-intensive workloads. Fully exploiting this is difficult: the APU uses a 32,768-element vector processor with column-wise integrated compute and storage, offering TB/s on-chip bandwidth but limiting memory access within a vector register (VR). For instance, reductions across a VR are unsupported, and intra-VR group operations are about 10 \times slower than inter-VR operations. Figure 2 . Performance of various matrix multiplication kernels on GSI APU, w/o data movement and data layout optimizations. Figure 2 shows a roofline model of different matrix multiplication kernels on the GSI APU 1 1 1 The peak computational bound is profiled for 16-bit unsigned multiplication and accumulation operations. . The baseline approach, implementing a vectorized inner-product algorithm, does not account for data movement or layout overheads, resulting in suboptimal performance. However, with tailored data optimizations, performance approaches the compute roof with higher operational intensity. This observation highlights a broader insight about compute-in-SRAM devices: despite performing computation directly within memory, these systems can still be bottlenecked by memory bandwidth if data movement is not carefully managed. To further analyze this issue, we develop an analytical framework that exposes the underlying performance limits. In this work, we propose three key optimizations to realize the potential of the compute-in-SRAM systems: communication-aware reduction mapping, coalesced DMA operations, and broadcast-friendly data layouts. Furthermore, we evaluate compute-in-SRAM systems for Retrieval-Augmented Generation (RAG) in large language model (LLM) inference, demonstrating its suitability for this workload. We also compare its performance and energy efficiency against CPU and GPU to highlight its advantages. Our contributions are as follows: • We present the first comprehensive evaluation of a commercial compute-in-SRAM device using realistic workloads. Specifically, we assess the GSI APU—a commercial instance of a general-purpose compute-in-SRAM device—using the Phoenix benchmark, matrix multiplication, and retrieval-augmented generation (RAG) workloads. We compare its performance and energy efficiency against established architectures, including an Intel Xeon Gold CPU and an NVIDIA A6000 GPU. • We develop a flexible analytical