PIPO: Pipelined Offloading for Efficient Inference on Consumer Devices
Yangyijian Liu, Jun Li, Wu-Jun Li · 2025
Read PIPO: Pipelined Offloading for Efficient Inference on Consumer Devices by Yangyijian Liu, Jun Li, Wu-Jun Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-03664.txt.
Introduction
Yangyijian Liu, Jun Li, and Wu-Jun Li proposed PIPO in 2025, a way to run large language models on small home devices. These models need more memory than a typical laptop graphics card holds. Moving parts of the model in and out can help, but the card then sits idle much of the time. Their design pipelines that offloading in fine steps.
PIPO pairs the pipeline with tuned data transfer and compute steps, so work overlaps instead of waiting. In tests, card use rose from below forty percent to over ninety percent. Throughput reached up to about three times that of a strong baseline system. The tests ran on a laptop with a modest card holding six gigabytes of memory. The result points to faster local AI on consumer hardware.
Catalog record
| CREATOR | Yangyijian Liu, Jun Li, Wu-Jun Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-03664
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PIPO: Pipelined Offloading for Efficient Inference on Consumer Devices Yangyijian Liu, Jun Li, Wu-Jun Li Abstract The high memory and computation demand of large language models (LLMs) makes them challenging to be deployed on consumer devices due to limited GPU memory. Offloading can mitigate the memory constraint but often suffers from low GPU utilization, leading to low inference efficiency. In this work, we propose a novel framework, called pip elined o ffloading (PIPO), for efficient inference on consumer devices. PIPO designs a fine-grained offloading pipeline, complemented with optimized data transfer and computation, to achieve high concurrency and efficient scheduling for inference. Experimental results show that PIPO can outperform existing methods. In particular, compared with state-of-the-art baseline, PIPO increases GPU utilization from below 40% to over 90% and achieves up to 3.1× higher throughput, running on a laptop equipped with a RTX3060 GPU of 6GB memory. 1 Introduction Large language models (LLMs) have demonstrated strong capabilities in many applications, such as writing, conversation and code generation, in recent years [ 1 , 2 ] . Due to the increasing demand for privacy preservation [ 3 ] and reducing deployment cost [ 4 ] , inference of LLMs on local devices (typically consumer devices) has been more and more important. However, the high memory and computation demand of LLMs makes them difficult to be deployed on consumer devices such as personal computers (PCs), since these devices typically use GPUs with limited memory. For example, the models with 7B or 30B parameters require 15GB to tens of GB of memory for storing model weights, which may be fitted on RTX3090 with 24GB memory but cannot be fitted on other RTX30 series with 6-12GB memory. Moreover, the Key-Value (KV) cache, essential for optimizing attention computations, grows proportionally with sequence length and batch size, further exacerbating the memory insufficiency problem. To achieve the goal of fitting LLMs into GPUs with limited memory, three techniques have been widely used, including quantization [ 5 , 6 , 7 ] , sparsification [ 8 , 9 , 10 ] and offloading [ 11 , 12 ] . Quantization uses lower precision of data and thus compresses the model. Sparsification improves efficiency by pruning and removing some weights and KV-cache. Offloading transfers weights and KV-cache from GPU memory to CPU memory or disk, for overcoming GPU memory limit. Although the quantization methods can decrease the model size by 4 \times with 4-bit precision (assuming the original value is of 16 bits), it remains insufficient for deploying a 30B model after quantization on a GPU with 8GB memory or less. Sparsification can greatly decrease the model size by up to 80%, but it suffers from non-negligible accuracy degradation. Hence, offloading has become a necessary strategy for deploying LLMs, especially LLMs with relatively large size, on consumer devices. Offloading adopts multi-level memory to schedule the excessive memory demands during the LLMs’ inference. For example, llama.cpp [ 12 ] distributes model layers between the CPU and GPU, leveraging both for inference. However, this hybrid inference approach is hindered by CPU’s limited computational ability on consumer devices, resulting in high inference latency. FlexGen [ 4 ] focuses on GPU-centric computation, utilizing a three-level memory hierarchy that includes GPU memory, CPU memory and disk. During LLMs’ inference, activation, weight and KV-cache are scheduled to be transferred between these memory levels, overlapping with the computation on the GPU. However, there still existed two issues in existing offloading frameworks: insufficient inference concurrency and underutilization of disks’ bandwidth. Insufficient Inference Concurrency: While LLMs are often viewed as memory-bound due to the frequent matrix multiplication operations with low arithmetic intensity, offloading exacerbates this issue, causing high data transfer latency. For instance, even with overlap techniques applied in the FlexGen framework, its coarse-grained control and low concurrency result in substantial inefficiency. In the case of OPT-30B with CPU-offloading, over 90% of the inference time is spent on data transfer, with GPU computation accounting for only about 5%. This insufficient inference concurrency leads to significant GPU idle time (low GPU utilization) and severely limits throughput. Underutilization of Disks’ Bandwidth: Although many of the existing offloading frameworks like FlexGen support disk-offloading, they primarily rely on CPU-offloading, requiring up to 200GB of CPU memory to offload the entire model. This far exceeds the typical memory capacity of consumer devices and neglects the potential of disks in LLM inference. In contrast, NVMe M.2 solid-state Drives (SSDs) offer significantly higher bandwidth than hard disk drives (HDDs), providing a more practical solution for offloading on consumer devices. Unfortunately, existing methods do not effectively utilize SSD bandwidth and pay insufficient attention to disk-based optimizations, leading to inefficient performance in disk-offloading inference. In this work, we propose a novel framework, called pip elined o ffloading (PIPO), for efficient inference on consumer devices by addressing the above two issues. The main contributions of PIPO are outlined as follows: • PIPO designs a fine-grained offloading pipeline with high concurrency that optimally balances memory usage and inference efficiency, supporting larger models on limited GPU memory. • PIPO emphasizes the crucial role of NVMe SSDs in offloading on consumer devices and designs a comprehensive suite to optimize data transfer speed between GPU memory, CPU memory and disk. • Experimental results show that PIPO can outperform existing methods. In particular, compared with state-of-the-art baseline, PIPO increases GPU utilization from below 40% to over 90% and achieves up to 3.1 \times higher throughput, running on a laptop equipped with a RTX3060 GPU of 6GB memory. 2 Related Work With the growing interest in deploying LLMs on local devices, inference frameworks based on offloading strategies have gained significant attention and research focus. DeepSpeed [ 11 ] and HuggingFace Accelerate [ 13 ] are among the first systems to integrate offloading into inference. They directly adopt offloading techniques from training, without in-depth study of the distinct characteristics of the inference phase. FlexGen [ 4 ] proposes an efficient offloading strategy through a search-based algorithm, optimizing data placement and overlapping computations with I/O operations. Powerinfer [ 9 ] and LLM in a flash [ 14 ] both utilize sparsification by keeping hot neurons on the GPU while offloading the rest weights to the CPU. During inference, they dynamically fetch necessary neurons from storage to device’s memory for computation. Although this improves the inference speed, it still suffers from accuracy degradation and high CPU memory overhead. While these offloading-based frameworks explore offloading strategies, they fall short of a thorough analysis of the associated performance overhead. FlexGen attempts to enhance efficiency through overlapping, but it still falls short of fully leveraging concurrency in the inference process. In addition to the strategy of loading data to the GPU for computation during inference, an alternative approach is to distribute computations across both the GPU and CPU. llama.cpp [ 12 ] implements this by partitioning the model at the Transformer layer level, storing some weights on the GPU while offloading the rest to the CPU. The CPU processes its assigned layers first and then transfers the results to the GPU for further computation. While this method reduces PCIe data transfer, it results in substantial GPU idle time and slows down inference. From an algorithmic perspective, quantization [ 7 , 5 ,