Hemlet: A Heterogeneous Compute-in-Memory Chiplet Architecture for Vision Transformers with Group-Level Parallelism DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Hardware Emerging architectures CCS: Computing methodologies Computer vision CCS: Computer systems organization Heterogeneous
Cong Wang, Zexin Fu, Jiayi Huang, Shanshi Huang · 2025
Read Hemlet: A Heterogeneous Compute-in-Memory Chiplet Architecture for Vision Transformers with Group-Level Parallelism DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Hardware Emerging architectures CCS: Computing methodologies Computer vision CCS: Computer systems organization Heterogeneous by Cong Wang, Zexin Fu, Jiayi Huang, Shanshi Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15397.txt.
Introduction
Cong Wang, Zexin Fu, Jiayi Huang, and Shanshi Huang present Hemlet, a computer chip design built to run vision transformer models with less energy. Vision transformers lead on tasks like naming objects in images, but they need much memory and computing power. Hemlet splits the work across small linked chips, called chiplets, that compute inside memory itself.
The design mixes analog and digital memory chips with a chip that handles data between steps. Work is shared across groups of chips in parallel, which cuts the costly talk between chips. In tests, Hemlet ran several times faster than other layouts and reached high speed at good energy use, making large vision models easier to deploy.
Catalog record
| CREATOR | Cong Wang, Zexin Fu, Jiayi Huang, Shanshi Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-15397
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hemlet: A Heterogeneous Compute-in-Memory Chiplet Architecture for Vision Transformers with Group-Level Parallelism DOI: XXXXXXX.XXXXXXX ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Hardware Emerging architectures CCS: Computing methodologies Computer vision CCS: Computer systems organization Heterogeneous (hybrid) systems Cong Wang, Zexin Fu, Jiayi Huang, Shanshi Huang Abstract . Vision Transformers (ViTs) have established new performance benchmarks in vision tasks such as image recognition and object detection. However, these advancements come with significant demands for memory and computational resources, presenting challenges for hardware deployment. Heterogeneous compute-in-memory (CIM) accelerators have emerged as a promising solution for enabling energy-efficient deployment of ViTs. Despite this potential, monolithic CIM-based designs face scalability issues due to the size limitations of a single chip. To address this challenge, emerging chiplet-based techniques offer a more scalable alternative. However, chiplet designs come with their own costs, as they introduce expensive communication, which can hinder improvements in throughput. This work introduces Hemlet—a heterogeneous CIM chiplet system designed to accelerate ViT. Hemlet facilitates flexible resource scaling through the integration of heterogeneous analog CIM (ACIM), digital CIM (DCIM), and Intermediate Data Process (IDP) chiplets. To improve throughput while reducing communication overhead, it employs a group-level parallelism (GLP) mapping strategy and system-level dataflow optimization, achieving speedups ranging from 1.89× to 4.47× across various hardware configurations within the chiplet system. Our evaluation results demonstrate that Hemlet can achieve a throughput of 9.24 TOPS with an energy efficiency of 4.98 TOPS/W. 1. Introduction The Vision Transformer (ViT) ( 16 ) has emerged as a foundational model that replaces convolutional neural network (CNN) with Transformer blocks, achieving state-of-the-art performance in computer vision. However, this improvement in performance comes with significantly higher computational demands, primarily driven by massive vector-matrix multiplications (VMMs). In addition to its computational intensity, ViT models are characterized by larger parameter sizes and require substantial frequent memory access to weights during inference. This frequent data transfer limits both throughput and energy efficiency on traditional von Neumann architectures, presenting a challenge commonly referred to as the "memory wall" bottleneck. The compute-in-memory (CIM) paradigm ( 11 ; 39 ; 45 ; 55 ) addresses this bottleneck by performing VMMs directly within memory arrays, thereby reducing data movement and enhancing overall efficiency. Under the CIM paradigm, heterogeneous architectures that combine emerging non-volatile memory (eNVM)-based and SRAM-based CIM designs have been widely explored to address the diverse computation and storage demands of Transformer workloads ( 44 ; 32 ; 27 ; 20 ) . This heterogeneous design is primarily driven by the intrinsic trade-offs between different memory technologies. On one hand, eNVM devices, such as resistive random access memory (RRAM), offer high storage density and high energy efficiency ( 53 ; 21 ) . However, their limited write endurance and high write overheads pose challenges for handling dynamically generated VMMs within multi-head attention (MHA) ( 7 ; 56 ; 2 ) . On the other hand, SRAM-based CIM designs provide faster access and better write endurance, but suffer from lower storage density and higher area overhead. To leverage the complementary strengths of these memory technologies, heterogeneous CIM systems typically employ eNVM-based CIM for static VMMs, which utilize fixed pretrained weights. Meanwhile, SRAM-based CIM is used to handle dynamic VMMs that require frequent operand updates during inference. This division of roles allows for efficient support of both static and runtime-generated computations in Transformer inference. While the heterogeneous design can effectively address the demands of Transformers and maximize performance by statically storing all pretrained weights on-chip, the large parameter sizes of these models present significant challenges. Specifically, storing all pretrained weights in eNVM-based arrays substantially increases chip area, leading to considerable issues related to fabrication cost and yield ( 17 ) . Furthermore, these designs are often optimized for specific workloads (e.g., ViTs, or customized Transformer variants). As model architectures evolve, these tightly coupled designs necessitate complete redesign, verification, and fabrication cycles, resulting in high recurring engineering costs and extended development timelines. To address the challenges posed by monolithic heterogeneous CIM architectures, chiplet-based design emerges as a promising solution. By integrating multiple chiplets within a single package, these systems enhance both modularity and scalability ( 40 ; 6 ) . Recent studies have started to investigate the potential of CIM chiplet platforms for accelerating DNN workloads ( 26 ; 25 ; 41 ; 15 ; 12 ) . However, the scalability offered by chiplet-based systems comes at a cost. Chiplets connected via network-on-package (NoP) face greater data transmission overhead compared to network-on-chip (NoC) on a single chip ( 4 ; 42 ; 6 ) . Therefore, optimizing the dataflow to reduce transmission overhead is crucial to maintaining performance in chiplet-based systems. Moreover, we observed that the conventional layer-wise mapping method used for ACIM fails to fully utilize the available hardware resources in real implementations, thereby limiting throughput. This presents an opportunity to compensate for the performance degradation caused by chiplet partitioning. In this work, we introduce Hemlet—a heterogeneous CIM chiplet-based system designed for efficient inference of ViTs. Hemlet integrates a group-level parallelism (GLP) weight mapping strategy with optimized inter- and intra-chiplet dataflow. Our key contributions are summarized as follows: (1) We propose Hemlet , a heterogeneous CIM chiplet-based architecture, integrating three types of chiplets: Analog CIM (ACIM), Digital CIM (DCIM), and Intermediate Data Process (IDP). By decoupling functionality across chiplets while maintaining intra-chiplet uniformity, Hemlet supports both architectural scalability and manufacturing efficiency. (2) A novel group-level parallelism (GLP) mapping method is proposed. In this approach, ACIM columns that share the same ADC are defined as a "Group" and parallelism among them is exploited to fully utilize the hardware resources. To facilitate efficient mapping of ViT models to Groups, we introduce an intermediate representation called the GLP_LayerSet , upon which we have developed a four-stage mapping procedure. (3) System-level dataflow is optimized to minimize inter-chiplet communication. A fine-grained pipeline is utilized for the communication and computation of static VMMs, effectively masking inter-chiplet communication between ACIM and IDP. Furthermore, the DCIM chiplet architecture is co-optimized with blocked dynamic VMMs, eliminating global buffer access across chiplets. These designs significantly enhance throughput in Hemlet’s heterogeneous chiplet architecture. (4) We conduct comprehensive evaluations of Hemlet across a range of ViT workloads and system configurations. The results show that GLP significantly improves ACIM throughput by maximizing hardware utilization. Additionally, the proposed Hemlet architecture, which combines GLP with system-level dataflow optimization, consistently achieves speedups across diverse hardware settings. Moreover, Hemlet demonstrates competitive performance compared to state-of-the-art accelerators while offering enhanced flexibility and scalability. The remainder of this paper is organized as follows: Section 2 introduces the preliminaries of compute-in-memory and chiplet-based architectures. Section 3 presents the proposed Hemlet architecture, followed by the GLP weight mapping strategy in Section 4 . Section 5 details the system-level dataflow optimization. Some evaluation results are presented in Section 6 and a conclusion is drawn in Section 7 . 2. Preliminary This section introduces background on CIM technology, including ACIM, DCIM, and their heterogeneous integration for Transformer acceleration, as well as the basics of chiplet-based architecture. 2.1. Compute-in-Memory Figure 1. Architecture of (a) Analog CIM and (b) Digital CIM Figure 2. Heterogeneous CIM for Transformer Computation Analog Compute-in-Memory (ACIM): Conductance-based ACIM macros perform VMMs by leveraging the resistive properties of emerging non-volatile memory (eNVM) devices such as resistive