Patchwork : A Unified Framework for RAG Serving
Bodun Hu, Luis Pabon, Saurabh Agarwal, Aditya Akella · 2025
Read Patchwork : A Unified Framework for RAG Serving by Bodun Hu, Luis Pabon, Saurabh Agarwal, Aditya Akella (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07833.txt.
Introduction
Saurabh Agarwal and co-authors present Harmonia, a 2025 system for serving retrieval-backed chatbots at scale. Such systems fetch facts, then let a language model answer, but requests cross models, databases, and plain processors. Slow links anywhere in that chain can break reply-time goals.
Harmonia lets builders spell out custom flows, places each part where it runs best, and steers load live. A feedback loop ranks requests and scales parts up or down as traffic shifts. Across four test apps, it more than doubled throughput against trade rivals and cut missed time goals sharply. The work targets shops that run such bots in production.
Catalog record
| CREATOR | Bodun Hu, Luis Pabon, Saurabh Agarwal, Aditya Akella |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-07833
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Patchwork : A Unified Framework for RAG Serving Bodun Hu, Luis Pabon, Saurabh Agarwal, Aditya Akella Abstract . Retrieval Augmented Generation (RAG) has emerged as a new paradigm for enhancing Large Language Model reliability through integration with external knowledge sources. However, efficient deployment of these systems presents significant technical challenges due to their inherently heterogeneous computational pipelines comprising LLMs, databases, and specialized processing components. We introduce Patchwork , a comprehensive end-to-end RAG serving framework designed to address these efficiency bottlenecks. Patchwork ’s architecture offers three key innovations: First, it provides a flexible specification interface enabling users to implement custom RAG pipelines. Secondly, it deploys these pipelines as distributed inference systems while optimizing for the unique scalability characteristics of individual RAG components. Third, Patchwork incorporates an online scheduling mechanism that continuously monitors request load and execution progress, dynamically minimizing SLO violations through strategic request prioritization and resource auto-scaling. Our experimental evaluation across four distinct RAG implementations demonstrates that Patchwork delivers substantial performance improvements over commercial alternatives, achieving throughput gains exceeding 48% while simultaneously reducing SLO violations by ∼ \sim 24%. 1. Introduction With their rapid advancements, Large Language Models (LLMs) have become indispensable tools across a range of applications, including question answering ( yang2015wikiqa ; rajpurkar2016squad ) , code generation ( zhong2017seq2sql ; liang2021lyra ) , task automation ( handa2025economic ) , and search assistants. Despite major progress in underlying machine learning (ML) techniques, LLMs still face several fundamental limitations—such as hallucinations ( friel2023chainpoll ) , outdated knowledge ( lewis2020retrieval ) , and confinement to the snapshot of training data. To address these issues, Retrieval-Augmented Generation (RAG) systems ( lewis2020retrieval ) have emerged as a widely adopted solution, particularly for applications where factual accuracy is critical. Figure 1 provides an overview of a typical RAG architecture. As illustrated, a RAG system supplements an LLM with a knowledge base—often an external, indexed database—whose contents are retrieved and incorporated into user queries before being passed to the LLM. Some RAG variants ( ircot ; borgeaud2022improving ; jiang2023active ; trivedi2022interleaving ) go further by enabling recursive interactions between the LLM and the knowledge base, allowing iterative refinements of the response. This integration has been shown to significantly improve both the factual accuracy and generation quality of LLM outputs. Moreover, RAG systems offer a practical alternative to costly model retraining, providing a scalable and extensible mechanism for keeping LLMs up to date with new information. These advantages have spurred rapid research and real-world adoption of RAG techniques ( qian2024memorag ; hipporag ; zhang2024raft ; edge2024local ; martis2024luminirag ; fang2025kirag ) . Given the growing ubiquity of RAG systems, optimizing their performance is becoming increasingly critical. However, three key challenges make this far from straightforward: 1. Rapid Evolution of the Stack : The RAG ecosystem is evolving at a breakneck pace, with new components, execution strategies, and architectural patterns emerging frequently. This constant change compounds the complexity faced by users in designing and deploying RAG pipelines. More importantly, the resulting fragmentation—a patchwork of bespoke solutions—makes it difficult to develop and apply general, reusable performance management techniques. 2. Heterogeneity of Components : RAG consist of diverse components—including vector databases, query augmenters, and LLM serving engines—each with unique computational characteristics and scaling behaviors. This makes performance control via resource management and workload partitioning significantly more challenging than in monolithic LLM deployments. In this work we call this collection of diverse components a RAG pipeline. 3. Unpredictable Execution Time : Whereas conventional neural inference exhibits stable and predictable latency ( gujarati2020serving ; narayanan2020heterogeneity ) , RAG pipelines introduce wide variability. Query-dependent branching, conditional execution paths, and recursive calls to retrievers or LLMs cause highly unpredictable execution times, complicating RAG systems’ ability to meet tight service-level objectives (SLOs). Figure 1 . Overview of RAG Pipeline: The above figure shows a schematic of RAG pipeline with details in Section 2. RAG logic can vary dramatically from simple concatenation operation to a complicated execution logic with conditional and recursive operations. This unique combination of attributes is un-studied in the computer systems resource allocation space, making the problem we target novel and interesting. To this end, our work introduces Patchwork , a system designed for the scalable, resource-efficient deployment of RAG pipelines. Patchwork ’s ground-up design directly addresses the above challenges enabling robust, high-performance operation of modern RAG systems. The key insight powering Patchwork ’s design is that enabling efficient RAG systems while both embracing the above complexities and hiding them from users requires a high-level, cross-component approach. To this end, Patchwork is designed as an intent-driven end-to-end framework. Users simply specify the RAG pipeline. Patchwork then automatically orchestrates the entire pipeline, conducting holistic performance-driven component-level resource management, resource allocation, and sharing depending on component-specific scalability patterns. This is in contrast to prior RAG approaches ( ray2024ragserve ; shi2024enhancing ) that are point solutions – they optimize specific components of a RAG pipeline, and as such, they cannot offer improvements as the workload and component characteristics change. Specifically, Patchwork introduces three key ideas. First, to support rapidly evolving RAG workloads, Patchwork ’s light-weight client library provides an interface for users to specify their pipelines using a set of simple, high-level modular abstractions. Through this interface, pipelines can be implemented in standard Python, requiring users only to annotate their compute nodes using a lightweight decorator provided by Patchwork . The client library also allows users to easily create new RAG components or extend existing ones. Patchwork then automatically captures the full computational graph using the Python frame evaluation API (introduced in Python 3.6 via PEP 523 ( pep523 ) ). This enables deep inspection and orchestration of the pipeline without requiring changes to user logic. To further support the continual evolution of RAG systems, Patchwork also allows easy extension of these abstractions, enabling seamless integration of new components as they emerge. Additionally, Patchwork ’s interface allows users to write the high-level declarative execution pipeline, at time of deployment Patchwork automatically distributes this computation akin to prior data processing systems ( dean2008mapreduce ; zaharia2010spark ) . Next, to support heterogeneous resource requirements and scaling patterns of RAG components, the Patchwork scheduler offline component leverages the client library-provided computation graph to perform component-level resource allocation and batching . Here, Patchwork abstracts the details of a RAG system’s underlying components by modeling both problems using a novel max-flow formulation aimed at maximizing pipeline throughput. Finally, to minimize SLO violations despite unpredictable execution times, Patchwork scheduler’s online component continuously tracks the run time of requests being served, estimates the likelihood of a particular request violating the SLO, and, correspondingly adjusts scheduling and resource allocations in a performance-aware fashion . Specifically, upon observing that a request is likely to miss its SLO, the online component takes proactive measures including: (i) prioritizing such requests, (ii) temporarily pausing accepting new requests, and (iii) auto-scaling resources where possible. To show the efficacy of Patchwork , we implement four different RAG algorithms - and show that Patchwork can improve the throughput of RAG serving between 48% and 20 \times . Further, Patchwork also reduces SLO violation by up to 24%. In Summary with Patchwork we make following contributions - • To the best of our knowledge we provide the first end to end