RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU
Weiping Yu, Ningyi Liao, Siqiang Luo, Junfeng Liu · 2025
Read RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU by Weiping Yu, Ningyi Liao, Siqiang Luo, Junfeng Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15302.txt.
Introduction
Answer-first chatbots that look facts up before replying strain small computers: one graphics card must hold both the model and the knowledge base. Weiping Yu and co-authors built RAGDoll in 2025, a serving system that runs this retrieval-augmented setup on a single card.
Their insight is that lookup and writing make different demands, so running them in a single line wastes both. RAGDoll splits them into parallel pipelines, with joint memory placement and smart batch scheduling. Tests across hardware setups and model sizes showed up to 3.6 times lower average delay than a leading serial system. The design adapts itself as workloads change.
Catalog record
| CREATOR | Weiping Yu, Ningyi Liao, Siqiang Luo, Junfeng Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-15302
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RAGDoll: Efficient Offloading-based Online RAG System on a Single GPU Weiping Yu, Ningyi Liao, Siqiang Luo, Junfeng Liu Abstract . Retrieval-Augmented Generation (RAG) enhances large language model (LLM) generation quality by incorporating relevant external knowledge. However, deploying RAG on consumer-grade platforms is challenging due to limited memory and the increasing scale of both models and knowledge bases. In this work, we introduce RAGDoll , a resource-efficient, self-adaptive RAG serving system integrated with LLMs, specifically designed for resource-constrained platforms. RAGDoll exploits the insight that RAG retrieval and LLM generation impose different computational and memory demands, which in a traditional serial workflow result in substantial idle times and poor resource utilization. Based on this insight, RAGDoll decouples retrieval and generation into parallel pipelines, incorporating joint memory placement and dynamic batch scheduling strategies to optimize resource usage across diverse hardware devices and workloads. Extensive experiments demonstrate that RAGDoll adapts effectively to various hardware configurations and LLM scales, achieving up to 3.6 × \times speedup in average latency compared to serial RAG systems based on vLLM. DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 1. Introduction The emergence of large language models (LLMs) ( Brown et al., 2020 ; Achiam et al., 2023 ; Touvron et al., 2023 ) has transformed diverse aspects of people’s daily lives and production, fostering applications including programming assistants ( Chen et al., 2021 ; Github., 2022 ) and universal chatbots ( Google., 2023 ; OpenAI., 2022 ) . To ensure the quality and relevance of LLM responses in context-rich scenarios, Retrieval Augmented Generation (RAG) ( Asai et al., 2023 ) has become popular for invoking external knowledge bases, such as local and private data in companies ( Chen et al., 2024 ; Luo et al., 2023 ; Bayarri-Planas et al., 2025 ; Morris et al., 2023 ) , into LLM inference. Local RAG utilizations, typically serving a small number of concurrent queries, focus on responding with low overall latency across these batches of data ( Fan et al., 2025 ; Prabhune and Berndt, 2024 ; Wang et al., 2024b ) . In general, a RAG workflow first retrieves data from the knowledge database according to user requests, then generates responses by the LLM based on the extracted information. The integration of retrieval-based techniques, however, poses new challenges for comprehensive performance enhancements, especially as RAG systems are commonly deployed in environments with limited computational resources. This issue stems from the inherent tension between retrieval and generation components, which exhibit different characteristics in terms of computation needs and memory usage patterns. For example, while the retrieval operation benefits from higher database throughput through batch processing with larger batch sizes ( Milvus., 2022 ; Chroma., 2025 ) , the generation operation usually demands a limited batch size to ensure low latency considering LLM tensor placement ( Song et al., 2024 ; Kwon et al., 2023 ; Sheng et al., 2023 ) . Figure 1 . Representative techniques related to RAG serving, depicted by corresponding tasks and design objectives. In Figure 1 , we present existing studies representing LLM serving, RAG computation, and database retrieval optimization techniques from two dimensions: their intended tasks focusing on retrieval or generation processes, and the performance bottleneck in relation to computation and storage resources. Since most of these techniques focus only on a single component or design objective within the RAG system, directly applying them for RAG deployment on resource-constrained platforms leads to issues from two perspectives. First, the RAG workflow features various data placements across devices , especially concerning the GPU-CPU-disk storage hierarchy ( Sheng et al., 2023 ) . LLM serving frameworks usually leverage offloading techniques ( Aminabadi et al., 2022 ; HuggingFace Accelerate., 2022 ; Kwon et al., 2023 ; Song et al., 2024 ) in consumer-grade environments, transferring idle data to RAM and disk to save GPU memory. Vector databases used for knowledge retrieval also actively maintain partial or full-scale data in memory ( Douze et al., 2024 ; Aumüller et al., 2020 ; Malkov and Yashunin, 2018 ) or on disk ( Wang et al., 2021 ; Chroma., 2025 ; Jayaram Subramanya et al., 2019 ) . While both components tend to allocate more main memory for better performance, the available memory capacity is relatively limited compared to the sizes of the LLM and database. Excessive overhead risks memory thrashing or even exhaustion, resulting in impeded response latency. Second, the processing and transmission of data over time need to be carefully designed. The RAG pipeline manipulates a batch of data to be first computed from knowledge embeddings ( Lewis et al., 2020 ; Gao et al., 2023 ) by the CPU during retrieval, and then transferred to the GPU for generation. Additionally, the offloading strategy periodically relocates LLM tensors between the GPU and CPU. These operations exhibit intricate dependencies. Current studies on RAG serving enhancements ( Jin et al., 2024 ; Yao et al., 2024 ) predominantly employ a straightforward serial processing strategy, which relies on the smooth execution of every phase. However, our evaluation reveals that overall latency is undermined by insufficient data utilization when computation and memory resources are constrained. Hence, a comprehensive pipeline is necessary to minimize redundant data processing, waiting, and transmission time, while ensuring concurrent utilization of various devices. In this paper, we propose RAGDoll , a system for RAG deployment that integrates LLM and database components on a single consumer-grade GPU. As illustrated in Figure 1 , RAGDoll encompasses a wide design space for collaboratively enhancing both computation and storage utilization, exploiting the benefits of different serving models adaptively toward lower latency. Specifically, RAGDoll highlights three key designs for addressing the above challenges. To effectively utilize storage across devices, RAGDoll introduces an innovative joint memory placement scheme integrating both retrieval and generation components. The system strategically optimizes placement across the entire storage hierarchy involving GPU, CPU, and disk, accounting for their distinct capacities and access speeds. This comprehensive maneuver improves device usage, achieving 2\times improvement in average CPU utilization in practice compared to conventional serial RAG implementations. The dominant memory overhead elements, including database partitions and LLM tensors, can be coordinated through adjusting pivotal configuration parameters in RAGDoll . To enhance data utilization, RAGDoll proposes a novel multi-pipeline RAG integration that decouples CPU-bound retrieval and GPU-bound generation stages. This parallel execution on different computational devices effectively reduces the idle time from 80% to 30% of the overall latency in the RAG workflow. Furthermore, the fine-grained data flow manipulation loosens the component interdependencies, enabling a broader range of pipeline configurations. Lastly, the joint and pipelined designs bring about the adaptive scheduling ability of RAGDoll for serving scenarios where static schemes fall short. Thanks to the dedicated data placement and pipeline execution modules, the impact on system workload can be effectively adjusted through a few performance-related parameters, especially batch size and resource allocation. The configurations in RAGDoll serving are designed to be dynamically accommodated to the status of incoming queries and device utilization. To this end, RAGDoll can adapt to different serving requirements by selecting the most suitable scheme dynamically, achieving peak latency reductions of up to 80%. We implement a RAGDoll