SLIM: A Heterogeneous Accelerator for Edge Inference of Sparse Large Language Model via Adaptive Thresholding Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 CCS: Hardware Memory and dense s

Weihong Xu, Haein Choi, Po-kai Hsu, Shimeng Yu, Tajana Rosing · 2025

Read SLIM: A Heterogeneous Accelerator for Edge Inference of Sparse Large Language Model via Adaptive Thresholding Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 CCS: Hardware Memory and dense s by Weihong Xu, Haein Choi, Po-kai Hsu, Shimeng Yu, Tajana Rosing (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-09201.txt.

Introduction

Weihong Xu and co-authors present a 2025 chip design for running large language models on small edge devices. Full models are too big and memory-hungry for such hardware, yet much of their math is sparse: most neurons sit idle on any given input. Their system, SLIM, sets thresholds on the fly so only active neurons are fetched, which slashes data movement.

The hardware splits the work by layer type. Feed-forward weights live in dense flash memory with compute beside them, while attention runs in memory-side units. Against graphics card setups, the design ran thirteen to eighteen times faster than flash-based systems and used about a tenth of the energy of memory-based ones, with small accuracy loss.

Catalog record

CREATORWeihong Xu, Haein Choi, Po-kai Hsu, Shimeng Yu, Tajana Rosing
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-09201

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SLIM: A Heterogeneous Accelerator for Edge Inference of Sparse Large Language Model via Adaptive Thresholding Conference: Make sure to enter the correct conference title from your rights confirmation emai; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/18/06 CCS: Hardware Memory and dense storage CCS: Computer systems organization Embedded systems CCS: Computing methodologies Artificial intelligence Weihong Xu, Haein Choi, Po-kai Hsu, Shimeng Yu, Tajana Rosing Abstract . Large language models (LLMs), composed of Transformer decoders, have demonstrated unparalleled proficiency in understanding and generating human language. However, efficient LLM inference on resource-constraint embedded devices remains a challenge because of the sheer model size and memory-intensive operations that arise from feedforward network (FFN) and multi-head attention (MHA) layers. Existing accelerations offload LLM inference to heterogeneous computing systems comprising expensive memory and processing units. However, recent studies show that most hardware resources are not used because LLM exhibits significant sparsity during inference. The sparsity of LLMs provides a good opportunity to perform memory-efficient inference. In this work, we propose SLIM, an algorithm and hardware co-design optimized for sparse LLM serving on the edge. SLIM exploits LLM’s sparsity by only fetching activated neurons to significantly reduce data movement. To this end, the efficient inference algorithm based on adaptive thresholding is proposed to support runtime configurable sparsity at the cost of negligible accuracy loss. Then, we present the SLIM heterogeneous hardware architecture that combines the best of both near-storage processing (NSP) and processing-in-memory (PIM). SLIM stores FFN weights in high-density 3D NAND and computes FFN layers in NSP units, alleviating high memory requirements caused by FFN weights. The memory-intensive MHA with low arithmetic density is processed in the PIM module. By leveraging the inherent sparsity observed in LLM operations and integrating NSP with PIM techniques within SSDs, SLIM significantly reduces memory footprint, data movement, and energy consumption. Meanwhile, we present the software support for integrating design into existing SSD system. Our comprehensive analysis and system-level optimization demonstrate the effectiveness of our sparsity-tailored accelerator, offering 13-18 × \times throughput improvements over SSD-GPU system and 9-10 × \times better energy efficiency over DRAM-GPU system while maintaining low latency. 1. Introduction Large language models (LLMs) have become transformative tools in a wide range of applications, including natural language understanding ( Wang et al. 2023 ; Karanikolas et al. 2023 ) , content generation ( Kasneci et al. 2023 ; Sarsa et al. 2022 ; Swanson et al. 2021 ) , and healthcare ( Arora and Arora 2023 ; Cascella et al. 2023 ) . Recently, local and edge deployments of LLM inference are becoming increasingly important as they offer significant advantages in cost efficiency, latency reduction, and data privacy over centralized solutions. Performing inference directly on edge devices eliminates the need for network connections, reducing reliance on cloud infrastructure and thereby lowering operational costs. Additionally, edge inference minimizes latency by processing data closer to the source. This allows for faster response for real-time applications in fields like autonomous systems and healthcare. Moreover, as data privacy concerns grow, edge LLM inference provides a secure solution by keeping sensitive information on the device, reducing the risks associated with transmitting data over external networks. These benefits make edge-based LLM inference not only practical but essential for scalable and secure AI integration across various industries. LLMs’ remarkable capabilities in generating high-quality content are primarily attributed to the scaling of model sizes to billions of parameters, which enables them to store and process vast amounts of knowledge across diverse domains. The model scaling not only improves linguistic fluency and contextual understanding but also enhances the models’ ability to handle complex, multi-step reasoning and nuanced information across specialized fields. The sheer size of LLM presents significant computational challenges for low-latency and low-cost inference during the deployment phase, because LLM inference demands immense hardware resources and prolonged inference times. First, LLM storage exhibits a prohibitive memory footprint. State-of-the-art (SoA) LLMs ( Touvron et al. 2023 ; Zhang et al. 2022 ; Jiang et al. 2024 ; Team 2024 ) consume over 10GB to 100GB memory space to store the billion-scale parameters. The required memory easily exceeds the capacity of commodity hardware devices, such as graphics processing unit (GPU). Second, LLM inference is bottlenecked by the sequential and memory-bound operations from multi-head attention (MHA) and feedforward network (FFN) with low arithmetic intensity ( Park et al. 2024 ; Heo et al. 2024 ) . LLM’s expensive memory footprint and sequential execution not only increase the overall inference latency, but also limit the achievable throughput. Existing LLM inference is typically served online using server-grade GPUs and high-bandwidth memories. The average inference cost and latency are amortized by allocating large-volume user requests to large-scale hardware facilities ( Miao et al. 2024 ; Park et al. 2024 ; Heo et al. 2024 ) optimized for LLM, thereby delivering satisfactory service quality and throughput. However, the aforementioned solutions relying on high-performance GPUs and memories at scale are not viable for the edge deployment of LLM. This is because edge devices with limited resources are sensitive to deployment cost and hardware complexity. The expensive facilities are difficult to accommodate on edge scenarios. Consequently, we need a more accessible and cost-effective edge deployment solution of LLMs. To this end, SoA designs perform joint optimizations for the algorithm and hardware components. For the algorithm, recent studies ( Liu et al. 2023 ; Song et al. 2025 ; Song et al. 2024 ) show that LLM inference exhibits significant redundancy. For example, LLM inference exhibits a high degree of sparsity, leading to wasted memory access as substantial data movement involves inactive components. Zero skipping ( Liu et al. 2023 ; Mirzadeh et al. 2024 ; Song et al. 2024 ; Hwang et al. 2024 ; Song et al. 2025 ) and model pruning ( Wang et al. 2025 ; Yuan et al. 2023 ; Yu and Wu 2023 ; Chen et al. 2021 ; Hsu et al. 2022 ) are the two commonly adopted algorithm optimizations to reduce memory footprint and energy consumption. But these methods show severe accuracy loss, limited flexibility, or high fine-tuning cost (see Section 3 ). From the hardware perspective, SoA LLM accelerators ( Park et al. 2024 ; Heo et al. 2024 ; Sheng et al. 2023 ; Song et al. 2024 ; Alizadeh et al. 2024 ; Hwang et al. 2024 ; Huang et al. 2025 ) exploit heterogeneous computing to mitigate the high-cost hardware by offloading different LLM modules to cheaper hardware, including DRAM/SSDs combined with GPUs ( Park et al. 2024 ; Heo et al. 2024 ; Sheng et al. 2023 ; Song et al. 2024 ; Alizadeh et al. 2024 ; Hwang et al. 2024 ) or CPU-FPGA system ( Huang et al. 2025 ) . The heterogeneous designs effectively handle the large model size and memory-intensive operations. Nevertheless, according to our analysis in Section 3 , these designs suffer from slow cold inference ( Yi et al. 2023 ) and low throughput due to the limited PCIe bandwidth between SSD/DRAM and GPU. Hence, this necessitates a cost-effective LLM solution with improved latency. In this work, we mainly aim at developing a cost-effective solution for local and edge deployments of LLMs. These application scenarios pose distinct challenges,

Read the complete text