Sangam: A Chiplet-Based DRAM-PIM Accelerator with CXL Integration for LLM Inferencing
Khyati Kiyawat ∗ † {}^{*}\dagger , Zhenxing Fan ∗ † {}^{*}\dagger , Yasas Seneviratne † \dagger , Morteza Baradaran † \dagger , Akhil Shekar † \dagger , Zihan Xia ‡ \ddagger , Mingu Kang ‡ \ddagger , and Kevin Skadron † \dagger · 2025
Read Sangam: A Chiplet-Based DRAM-PIM Accelerator with CXL Integration for LLM Inferencing by Khyati Kiyawat ∗ † {}^{*}\dagger , Zhenxing Fan ∗ † {}^{*}\dagger , Yasas Seneviratne † \dagger , Morteza Baradaran † \dagger , Akhil Shekar † \dagger , Zihan Xia ‡ \ddagger , Mingu Kang ‡ \ddagger , and Kevin Skadron † \dagger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12286.txt.
Introduction
Khyati Kiyawat and seven co-authors present Sangam, a memory system designed to speed up inference for large language models. As models and context lengths grow, inference becomes limited by memory, especially the key-value cache used during decoding. Processing inside memory can help, but putting compute inside DRAM chips costs area and limits capacity.
Sangam splits logic and memory into separate chiplets, made in different processes and linked on an interposer with a standard interconnect. The logic chips reach the DRAM banks at high bandwidth and carry stronger processors, such as systolic arrays and fast buffers. The design targets the memory-bound matrix work that dominates decoding.
Catalog record
| CREATOR | Khyati Kiyawat ∗ † {}^{*}\dagger , Zhenxing Fan ∗ † {}^{*}\dagger , Yasas Seneviratne † \dagger , Morteza Baradaran † \dagger , Akhil Shekar † \dagger , Zihan Xia ‡ \ddagger , Mingu Kang ‡ \ddagger , and Kevin Skadron † \dagger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-12286
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sangam: A Chiplet-Based DRAM-PIM Accelerator with CXL Integration for LLM Inferencing Khyati Kiyawat ∗ † {}^{*}\dagger , Zhenxing Fan ∗ † {}^{*}\dagger , Yasas Seneviratne † \dagger , Morteza Baradaran † \dagger , Akhil Shekar † \dagger , Zihan Xia ‡ \ddagger , Mingu Kang ‡ \ddagger , and Kevin Skadron † \dagger Abstract Large Language Models (LLMs) are becoming increasingly data-intensive due to growing model sizes, and they are becoming memory-bound as the context length and, consequently, the key-value (KV) cache size increase. Inference, particularly the decoding phase, is dominated by memory-bound GEMV or flat GEMM operations with low operational intensity (OI), making it well-suited for processing-in-memory (PIM) approaches. However, existing in/near-memory solutions face critical limitations such as reduced memory capacity due to the high area cost of integrating processing elements (PEs) within DRAM chips, and limited PE capability due to the constraints of DRAM fabrication technology. This work presents a chiplet-based memory module that addresses these limitations by decoupling logic and memory into chiplets fabricated in heterogeneous technology nodes and connected via an interposer. The logic chiplets sustain high bandwidth access to the DRAM chiplets, which house the memory banks, and enable the integration of advanced processing components such as systolic arrays and SRAM-based buffers to accelerate memory-bound GEMM kernels, capabilities that were not feasible in prior PIM architectures. We propose Sangam, a CXL-attached PIM-chiplet based memory module that can either act as a drop-in replacement for GPUs or co-executes along side the GPUs. Sangam achieves speedup of 3.93, 4.22, 2.82x speedup in end-to-end query latency, 10.3, 9.5, 6.36x greater decoding throughput, and order of magnitude energy savings compared to an H100 GPU for varying input size, output length, and batch size on LLaMA 2-7B, Mistral-7B, and LLaMA 3-70B, respectively. I Introduction The global AI inference market is growing rapidly and generative AI inference is estimated to drive over 75% of AI compute and energy demand in U.S. data centers [ 1 ] . GenAI power demand in the US is projected to reach 800 TWh by 2030, increasing from 8TWh in 2024 [ 2 ] . Almost all leading large language models (LLMs), vision/speech models, and multimodal systems are based on the Transformer architecture [ 3 ] , with applications ranging from summarization and chatbots [ 4 ] to language-translation [ 5 ] and code generation [ 6 , 7 ] , among others. However, current state-of-the-art inference platforms rely heavily on costly, high-end servers equipped with multiple GPUs and high-bandwidth memory (HBM). Such hardware configurations are capital-intensive for datacenters and expensive to rent. An example is the NVIDIA’s H100 GPU which costs around $25k-30k and the rental costs for the same are estimated at $3 per hour [ 8 ] , not to mention the constrained availability of such hardware. Fig. 1 : LLaMA 2-7B inference kernel latency breakdown on one H100 GPU. Left and right figures show breakdown under varying batch sizes and input lengths when output length is 32 and 2048, respectively. (BS = Batch Size.) Prior works [ 9 , 10 ] find that, during the LLM token generation phase (predicting one output token at a time), inference is DRAM-bandwidth bound in most scenarios, and even prefill (processing the input prompt) is sometimes bandwidth bound (see Section II-B ). For example, GPU utilization during single-batch LLM inference workloads often remains below 20% [ 11 , 12 , 13 ] and never more than 40% as reported by [ 14 ] , indicating a substantial mismatch between provisioned compute and effective throughput, leading to poor utilization of an expensive resource such as the GPU. Techniques such as batching are employed to reduce memory accesses and improve processor utilization by enabling the reuse of weight matrices during GEMM operations. However, the achievable batch size is limited by memory capacity [ 12 ] , since the KV cache grows linearly with both batch size and sequence length, and by service-level objectives (SLOs) that impose minimum latency requirements, thereby restricting the extent of batching [ 15 ] . The inefficiency in state-of-the-art inference platforms primarily arises from memory bottlenecks, which manifest in two key dimensions: capacity and bandwidth. The escalating scale of GenAI inference underscores the need for a more balanced and cost-efficient design paradigm. Addressing this demand calls for research focused on two critical C hallenges: (C1) developing systems with scalable memory capacity, and (C2) minimizing over-provisioning of costly, underutilized hardware. One promising direction is to extend the functionality of an inference platform’s memory by augmenting it with processing-in-memory (PIM) capabilities that provide higher throughput for memory-bound computations by leveraging the very high bandwidth internal to the DRAM architecture. A compute-enabled memory system can potentially replace a GPU for much lower cost, or alternatively, enhance the GPU for a modest extra cost. For these reasons, Davies et al. [ 16 ] argue that DRAM-based inference solutions provide better capacity and bandwidth to achieve superior tokens/Watt and tokens/$ performance. Indeed, prior works [ 17 , 18 , 14 , 12 , 15 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 ] have demonstrated PIM as a potential solution to accelerate AI inference due to high internal bandwidth (orders of magnitude higher than a GPU with HBM at iso-capacity). However, most PIM approaches are limited in compute capability due to area and DRAM fabrication constraints. Thus, they targetted kernels with very low operational intensity (OI) of approximately 1 FLOP/byte. Supporting kernels with higher operational intensity (OI), such as matrix multiplication for prefill or batched decoding, requires more expensive logic units along with some buffering capacity to enable data reuse. Unfortunately, (C3.1) adding logic units and buffers near the memory cell array inside the DRAM die sacrifices memory capacity (reducing the memory density by 25%-75% [ 29 , 15 , 30 , 17 ] ). Furthermore, Figure 1 provides latency breakdown of each kernel contributing to inference as a function of batch size and input sequence length, with two example output lengths: 32 and 2048. When output length is 32 ( Figure 1 ), memory-bound GEMM kernels are the main contributor only when input is also small (32), while for larger batch sizes and larger input size (2048), both attention and compute-bound GEMM dominate inference latency. Notably, when the output length is 2048 (right figure), memory-bound GEMMs consistently account for a large share of latency across all batch sizes. This shows that end-to-end LLM inference relies heavily on GEMM kernels to support input processing and batched processing, with kernels computing >1 but <200 FLOPs/byte contributing the most to the end to end latency ( Section II-B ). These flat-GEMMs are memory-bound on GPUs, but inefficient when unrolled to a sequence of GEMV (matrix-vector) operations in DRAM PIM, which loses opportunities for data reuse through tiling, e.g. via systolic arrays. (C3.2) In addition, many prior PIM works [ 17 , 30 , 29 ] that put logic at the bank-level requires inefficient and expensive bank-to-bank data transfers. Adding any logic transistors in a DRAM process occupies 10 \times more area and runs at a slower clock compared to a logic process [ 31 ] . However, fabricating the