CIMPool: Scalable Neural Network Acceleration for Compute-In-Memory using Weight Pools

Shurui Li, Puneet Gupta · 2025

Read CIMPool: Scalable Neural Network Acceleration for Compute-In-Memory using Weight Pools by Shurui Li, Puneet Gupta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22044.txt.

Introduction

Shurui Li and Puneet Gupta present a way to fit larger neural networks into memory based chips in a 2025 paper. Computing inside memory avoids shuttling data back and forth, but common chip memory is small and cells are large. Their framework shares weights across the network through a weight pool, squeezing the model far more than standard precision cuts at the same accuracy.

The team co designed the squeezing method, hardware, and data flow so the needed shuffling costs almost nothing in area or speed. Results reach eight bit level accuracy with an effective half bit precision. Chip area for a known image network fell by more than sixty percent. Readers see how sharing weights could host far larger models on chip.

Catalog record

CREATORShurui Li, Puneet Gupta
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22044

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CIMPool: Scalable Neural Network Acceleration for Compute-In-Memory using Weight Pools Shurui Li, Puneet Gupta Abstract Compute-in-memory (CIM) based neural network accelerators offer a promising solution to the Von Neumann bottleneck by computing directly within memory arrays. However, SRAM CIM faces limitations in executing larger models due to its cell size and on-chip memory constraints. This work proposes CIMPool, a CIM-aware compression and acceleration framework that counters this limitation through a weight sharing-based compression technique, aptly named ‘Weight Pool,’ enabling significantly larger neural networks to be accommodated within on-chip memory constraints. This method minimizes the accuracy trade-off typically associated with parameter compression, allowing CIMPool to achieve a significantly larger compression ratio compared to the traditional quantization method with iso-accuracy. Furthermore, CIMPool co-optimizes the compression algorithm, hardware, and dataflow to efficiently implement the hardware permutation required by weight pool compression, with negligible area and throughput overhead. Empirical results demonstrate that CIMPool can achieve 8-bit level accuracy with an effective 0.5-bit precision, reduce chip area by 62.3% for ResNet-18, and enable the execution of an order of magnitude larger models for a given area budget in SRAM CIMs. When DRAM is used to store weights, CIMPool can reduce the total energy by 3.24 × 3.24\times compared to iso-accuracy traditional CIMs. I Introduction Compute-in-memory (CIM) based neural network accelerators have proliferated in recent years because of their unique advantage in alleviating the Von Neumann bottleneck. [ 1 , 2 ] . This is achieved by computing directly within memory arrays, significantly reducing data movement overhead. [ 3 , 4 , 5 , 6 ] Among various CIM accelerators, SRAM CIM is a promising candidate because of its technological maturity, high reliability, and low latency. Additionally, SRAM’s low write latency and high endurance enable the weights stored in the CIM to be rewritable at runtime, enhancing flexibility. However, SRAM CIM faces its own set of challenges. Its relatively large cell size makes area a limiting factor for most SRAM CIM accelerators, often hindering the execution of larger models. With the continual increase in the size of modern neural networks, improving the area efficiency of SRAM CIM is crucial to facilitate the execution of these larger networks. Due to the area limitations of SRAM CIM, a model that fits entirely on the CIM is typically too small to be useful without any additional compression techniques. Even with aggressive scaling, a large SRAM capacity is 16MB, which can at most contain 16M weights quantized to 8-bit. For example, this may hold an 8-bit ResNet18 but is too small to contain ResNet34 or larger. [ 7 ] On the other hand, if the network is too large to be contained solely on the CIM, additional performance costs are incurred to move the weights from the DRAM. Thus, if the neural network weights are not made to fit into a small area, the inference is slower and more energy-hungry due to the additional latency (and energy) costs of loading the weights from DRAM. Especially for low-batch size inference regimes, this wipes out the performance benefits created by CIM in the first place, namely the reduced data movement. While technology scaling and circuit-level optimizations are conventional solutions to this problem, our work adopts a novel approach, employing software and algorithm-level optimizations with co-optimization of CIM hardware. Traditional methods like quantization and compression effectively reduce the storage demands of neural networks but often overlook the underlying hardware. This oversight can lead to various overheads when implemented on CIMs, and some even make the implementation infeasible. To address these issues, we introduce CIMPool, a hardware-aware compression and acceleration framework tailored for scalable and efficient neural network acceleration in CIMs. The proposed framework is based on the weight pool concept [ 8 ] , a compression method through weight sharing, which compresses neural networks by converting their weights into a small pool (codebook) of weight vectors and the indices to the weight pool. This compression method fits CIM well because the codebook is fixed and is typically small enough to be stored within a single CIM array, which means the CIM array content can be fixed for the entire execution of the neural network. While the combination of weight pool and CIM seems promising, there are some challenges when applying weight pool to CIMs. One challenge is that the vector size of the original weight pool is just 8 for accuracy and implementation considerations, which is too small for CIM arrays that typically have a height a few times larger than 8, making the execution inefficient. However, naively increasing the vector size leads to a dramatic reduction in accuracy. To address this, CIMPool introduces a 1-bit error to mitigate the accuracy drop associated with the large vector size and further prunes the error to minimize the hardware overhead. Another main impediment to directly applying the weight pool compression method to CIM is the requirement of hardware permutation. In weight pool compression, adjacent weight vectors are assigned to different weight pool vectors (CIM columns). However, since the weight pool of the CIM is fixed, the CIM outputs need to be permuted back to their correct order, which can lead to significant hardware and latency overhead. CIMPool addresses this issue by leveraging SRAM CIM’s bit-serial processing property and co-optimizing the weight pool and hardware design. The main contributions of this work can be summarized as follows: • CIMPool compresses neural networks with underlying CIM in mind. Only two CIM arrays are required for processing an entire neural network. • By leveraging bit-serial computation and co-optimization of dataflow and weight pool, CIMPool can permute the CIM outputs in hardware with negligible overhead. • For the Food-101 dataset, CIMPool can achieve 8-bit level accuracy with 27.7 \times compression. • CIMPool can reduce chip area by 62.3% compared to iso-accuracy 4-bit CIMs. For a given area budget such as 100mm^{2} , CIMPool makes the execution of an order of magnitude larger models possible for SRAM CIMs. • When DRAM is used to store the weights, CIMPool can reduce the total energy by 3.35\times compared to iso-accuracy 4-bit CIMs. II Background and Motivation II-A SRAM CIM SRAM CIM emerges as a promising solution for energy-efficient neural network acceleration, merging computation and memory seamlessly to minimize both computation and data movement costs. An SRAM CIM array can perform a large vector-matrix multiplication in a single cycle, which is essential in neural network computations. In this setup, the neural network weights are mapped onto SRAM cells, while inputs are applied in parallel across the word lines. As inputs traverse through these word lines, each SRAM cell modulates the corresponding bit line’s current, effectively executing the multiplication. ADCs (Analog-to-digital converters) at the end of bit lines collect these currents, representing dot product results, and convert them into digital signals. Due to the single-bit nature of SRAM cells, multi-bit weights are stored across multiple bitlines. Multi-bit inputs are processed bit-serially, with partial sums accumulated through analog or digital shift-and-add units. In mapping a neural network layer to a CIM array, output channels (or filters in CNNs) are typically mapped to CIM columns, while input channels are mapped to rows. In CIMs, energy and area are usually dominated by the ADCs. To amortize the

Read the complete text