On-Device Qwen2.5: Efficient LLM Inference with Model Compression and Hardware Acceleration
Maoyang Xiang, Ramesh Fernando, Bo WangSingapore University of Technology and Design, Email: [email protected], [email protected], [email protected] · 2025
Read On-Device Qwen2.5: Efficient LLM Inference with Model Compression and Hardware Acceleration by Maoyang Xiang, Ramesh Fernando, Bo WangSingapore University of Technology and Design, Email: [email protected], [email protected], [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-17376.txt.
Introduction
Maoyang Xiang, Ramesh Fernando, and Bo Wang show how to run a small language model directly on an edge device. Their paper appeared in 2025. Large models strain small hardware through heavy computing, memory traffic, and power use. The team deployed a half-billion-parameter model on a chip board that pairs a modest processor with programmable logic. They shrank the model with a compression method that weighs which weights matter most.
Their setup splits work between the two halves of the board. Heavy operations go to the programmable logic, and light tasks stay on the processor. The model shrank by about 55 percent. Output speed rose to about five tokens per second, nearly double the baseline rate. The work shows that direct on-device language models are practical without a data center.
Catalog record
| CREATOR | Maoyang Xiang, Ramesh Fernando, Bo WangSingapore University of Technology and Design, Email: [email protected], [email protected], [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-17376
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On-Device Qwen2.5: Efficient LLM Inference with Model Compression and Hardware Acceleration Maoyang Xiang, Ramesh Fernando, Bo WangSingapore University of Technology and Design, Email: [email protected], [email protected], [email protected] Abstract Transformer-based Large Language Models (LLMs) have significantly advanced AI capabilities but pose considerable challenges for deployment on edge devices due to high computational demands, memory bandwidth constraints, and energy consumption. This paper addresses these challenges by presenting an efficient framework for deploying the Qwen2.5-0.5B model on the Xilinx Kria KV260 edge platform, a heterogeneous system integrating an ARM Cortex-A53 CPU with reconfigurable FPGA logic. Leveraging Activation-aware Weight Quantization (AWQ) with FPGA-accelerated execution pipelines, the proposed approach enhances both model compression rate and system throughput. Additionally, we propose a hybrid execution strategy that intelligently offloads compute-intensive operations to the FPGA while utilizing the CPU for lighter tasks, effectively balancing the computational workload and maximizing overall performance. Our framework achieves a model compression rate of 55.08% compared to the original model and produces output at a rate of 5.1 tokens per second, outperforming the baseline performance of 2.8 tokens per second. I Introduction Recent advancements in Large Language Models (LLMs) have spurred numerous opportunities across different domains [ 1 ] , attracting users from sectors such as healthcare [ 2 ] , robotics [ 3 ] , biomedical [ 4 ] , music [ 5 ] , etc. However, the growing adoption of these models has imposed immense computational demands, particularly for state-of-the-art models like GPT-4 [ 6 ] , DeepSeek-V3 [ 7 ] , and PaLM 2 [ 8 ] , which are typically confined to power-intensive data centers. In contrast, there is a rising need to deploy lightweight LLMs on edge devices to enable real-time responses in wireless-denied environments and enhance privacy protection [ 9 ] . In this work, we propose an efficient framework to accelerate on-device Qwen2.5-0.5B model inference. To implement it, we identify key challenges in deploying the model on resource-constrained edge devices and then employ the Xilinx Kria KV260 board, which incorporates the Kria K26 System-on-Module (SOM) (Fig. 1 ) as the target platform. Fig. 1 : Simplified hardware architecture of Xilinx Kria K26 system-on-module (SOM) adapted from [ 10 ] . Specifically, we identify significant challenges deteriorating the efficiency of model inference on Kria KV260 through a detailed analysis of the Qwen2.5-0.5B model, including both linear and non-linear operations. • Challenge 1: The limited memory capacity and bandwidth in the Programmable Logic (PL) side (e.g., Block RAMs, URAMs) restrict both the amount of model parameters that can be loaded and the efficiency of model loading, significantly impeding the deployment of on-device LLM inference. • Challenge 2: Matrix multiplications performed through Multiply-and-Accumulate (MAC) operations dominate the computational workload during inference, making them a performance bottleneck for LLM inference on edge devices. (a) Keeping 1% salient weights in FP16 (b) Scale the weights before quantization Fig. 2 : AWQ uncovers that (a) keeping only 1% of salient weights in FP16 can achieve a similar accuracy compared to the original model but not hardware friendly; (b) following activation awareness and performing per-channel scaling can protect the salient weights and reduce quantization error. The figure is adapted from work [ 11 ] . In this work, we address the challenges and achieve an excellent balance between accuracy, compression ratio, and tokens per second. The key contributions of our work are as follows. • We exploit a model compression technique at the software level to optimize memory bandwidth utilization and minimize data transfer between off-chip memory and processing elements. • We leverage the VFP4 floating point unit in ARM Cortex-A53 and parallelism of the FPGA fabric and successfully realize end-to-end inference of the Qwen2.5-0.5B model. • We further design an accelerator that can expedite matrix multiplications in the workload with piplined weight unpacking and dequantization. • We achieve a model compression rate of 55.1% and obtain an output rate of 5.1 tokens per second, surpassing the baseline performance of 2.8 tokens per second. II Background and Related Work This section provides an overview of the Qwen2.5 model architecture and the AWQ-based model compression technique. II-A The Qwen2.5 model and the Transformer Architecture The Transformer architecture [ 12 ] has become the foundation of modern large language models, primarily due to its encoder-decoder structure. In particular, the Qwen2.5 model series [ 13 , 14 ] adopts a decoder-only architecture that utilizes an auto-regressive approach for sequential text generation [ 15 ] . However, this inherently sequential decoding process poses significant computational challenges. Specifically, the decoder architecture in the Qwen2.5 model consists of a prefill stage and a decode stage. The prefill stage is characterized by highly parallel matrix-matrix operations while the decode stage is dominated by sequential matrix-vector multiplications. Consequently, accelerating matrix multiplications becomes crucial for optimizing compute efficiency and minimizing latency, particularly for applications requiring real-time inference capabilities. II-B Model compression with AWQ Activation-aware Weight Quantization (AWQ) [ 11 ] is a powerful model compression technique to significantly reduce memory footprint and lower memory bandwidth requirements. As illustrated in Fig. 2 , only 1% of the weights can have a significant impact on model performance. Therefore, carefully preserving those salient weights can maintain performance and minimize quantization error. Specifically, AWQ performs per-channel scaling based on activation distribution, so that the entire weight matrix can be quantized to lower precisions (e.g., INT4 and INT3) but still enjoy high performance. AWQ does not rely on model training or reconstruction. It utilizes a grouping mechanism to share the same scaling parameters with a default Group Size (GS) of 128. This grouping allows AWQ to accurately capture variations in weight distributions across different channels, effectively reducing quantization error and maintaining high model accuracy despite using low-precision representations. We believe such model compression is particularly advantageous in edge deployment scenarios, such as the KV260 platform which is constrained by a memory bandwidth of 19.2 GB/s. Our implementation extends AWQ’s benefits through a customized weight packing scheme, ensuring efficient delivery of the quantized weights (i.e., qweights), scaling factors, and zero values to the Processing Elements. This packing scheme facilitates on-the-fly dequantization, optimizing both performance and compute efficiency. II-C Related Work The utilization of FPGA to accelerate LLM inference has garnered significant interest from both academia and industry due to their exceptional energy efficiency, reconfigurability, and flexibility. A recent study [ 16 ] has introduced specialized spatial accelerators designed for efficient FPGA-based LLM inference, revealing the potential of FPGA platforms to provide customized, high-performance solutions. Particularly, the work in [ 17 ] has achieved substantial improvements in both performance and power efficiency after deploying the TinyLlama 1.1B model [ 18 ] on the Xilinx ZCU102 platform, demonstrating the feasibility of FPGA-based deployments in edge inference applications. III Proposed Software-Hardware Co-Optimization We perform an in-depth analysis of the computational load and delay associated with the Qwen2.5-0.5B model. Table I presents a latency breakdown of the Qwen2.5-0.5B model during inference when deployed on the PS of the KV260 platform, using the compiler optimizations to utilize all four ARM Cortex-A53 cores. It reveals that 91.6% of the inference time is dominated by MAC, which are the fundamental operations in matrix-matrix and matrix-vector multiplications. Note that the reported time in Table I accounts for