MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference
Joris Köster, Zixuan Liu, Siavash Khajavi, Zizhan Zheng · 2026
Read MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference by Joris Köster, Zixuan Liu, Siavash Khajavi, Zizhan Zheng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26557.txt.
Introduction
Köster and co-authors present MemBoost, a serving framework that cuts language model cost by reusing past answers. Real workloads repeat many near-duplicate queries across users and sessions. MemBoost lets a small model reuse stored answers and pull support from a growing memory. Hard or doubtful queries are escalated to a stronger model. Unlike standard retrieval setups, the design suits live chat with answer reuse and cost-aware routing. Simulated workloads show far fewer costly calls while answer quality stays near the strong baseline.
Catalog record
| CREATOR | Joris Köster, Zixuan Liu, Siavash Khajavi, Zizhan Zheng |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22613 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.26557
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference Joris Köster, Zixuan Liu, Siavash Khajavi, Zizhan Zheng Abstract Large Language Models (LLMs) deliver strong performance but incur high inference cost in real-world services, especially under workloads with repeated or near-duplicate queries across users and sessions. In this work, we propose MemBoost , a memory-boosted LLM serving framework that enables a lightweight model to reuse previously generated answers and retrieve relevant supporting information for cheap inference, while selectively escalating difficult or uncertain queries to a stronger model. Unlike standard retrieval-augmented generation, which primarily grounds a single response, MemBoost is designed for interactive settings by supporting answer reuse, continual memory growth, and cost-aware routing. Experiments across multiple models under simulated workloads show that MemBoost substantially reduces expensive large-model invocations and overall inference cost, while maintaining high answer quality comparable to the strong model baseline. 1 Introduction Large Language Models (LLMs) 14 ; 32 have demonstrated strong capabilities in natural language understanding, instruction following 6 ; 27 , coding 11 ; 25 , and decision-making 1 ; 37 . However, deploying frontier-scale models in real-world services remains expensive, as inference often requires multiple high-end GPUs, especially for long-form reasoning and explanation-heavy responses 16 ; 28 . These costs are amplified in production settings where many user queries are repeated or near duplicates, causing redundant computation. A widely adopted direction to mitigate these costs is to shift part of the workload to smaller models with retrieval-augmented generation (RAG) 17 ; 13 ; 4 . RAG allows a model to externalize knowledge into a searchable index and condition generation on retrieved evidence, reducing the need to store everything in parameters 15 ; 9 . For example, Atlas 15 demonstrates that the RAG model with far fewer parameters can be competitive with, and in some regimes outperform, much larger models on knowledge-intensive tasks. However, most existing RAG work primarily targets knowledge grounding, i.e., improving the answer to a single query by retrieving external documents 17 ; 13 . In contrast, interactive LLM services exhibit additional properties that are not addressed by standard RAG alone. First, many queries that are semantically equivalent are repeatedly requested across users and sessions. Recomputing these same answers wastes GPU time. This motivates semantic caching, which retrieves a previously generated response when a new query is semantically similar 12 ; 39 ; 21 . However, making this reliable requires careful control. Second, deployed LLMs continuously generate new high-quality answers. When these answers are valuable, they should be written back into the retrieval memory so that a semantic cache can serve future similar queries cheaply. However, this write-back mechanism raises practical questions, such as what/when to store it. Finally, although small models augmented with retrieval can be competitive on knowledge-intensive tasks, their capabilities remain limited on more challenging requests. A practical system should therefore enable the small model to escalate to a stronger model when retrieval is insufficient, aligning with recent work on routing across multiple LLMs 26 ; 40 ; 24 to balance quality and cost. These observations suggest a missing system-level paradigm: a unified approach that enables a small model to cheaply reuse supporting knowledge or prior answers from semantic memory, selectively defer to a stronger but costly model when retrieval is insufficient, and continuously write back useful new information into the retrieval memory, while explicitly balancing answer quality and inference cost. In this paper, we introduce MemBoost , a memory-boosted architecture with three components: (1) an Associative Memory Engine (AME) that performs fast semantic retrieval and supports write-back of newly generated answers; (2) a high-capability Large-LLM Oracle that provides an accurate fallback when memory is insufficient; and (3) a lightweight Meta Controller, which is a small LLM that composes the final response by either reusing from memory or routing the query to the oracle, and support write back mechanism for future reuse, thereby balancing quality and cost. Together, MemBoost turns inference into a “retrieve-or-escalate” decision problem, substantially reducing expensive large-model calls while preserving answer quality. Experiments on the MMLU-Pro dataset 36 under a simulated workload with repeated and near-duplicate queries show that MemBoost substantially reduces calls to the costly large LLM while largely preserving the answer quality of the oracle model. 2 MemBoost: Memory-Boosted LLM Serving Framework 2.1 Problem Setup Unlike standard single-turn question answering (QA) tasks studied in prior work, we consider a setting with continuous interaction between multiple users and an LLM service, where incoming queries may be exact duplicates or semantic near-duplicates over time. At each time step t , the system receives a user query x_{t} and produces an answer y_{t} . We use a ground-truth quality signal r(x_{t},y_{t}) to measure whether the response is helpful, correct, and aligned with the user’s intent. Our objective is to maximize the average response quality over a horizon T , \max\;\frac{1}{T}\sum_{t=1}^{T}r(x_{t},y_{t}), while minimizing the overall inference cost, as formalized in Section 2.3 . 2.2 Overview of MemBoost As discussed above, in production settings, repeatedly invoking a frontier model to answer semantically redundant queries wastes substantial compute. To mitigate this inefficiency while balancing quality and cost, we propose MemBoost, a memory-boosted LLM serving system composed of three components: Associative Memory Engine, Large-LLM Oracle and Meta Controller, as illustrated in Figure 1 . Figure 1: Overview of MemBoost. For each incoming query, the AME retrieves a small set of relevant memory entries. The MC then either composes an answer from the retrieved results or escalates to the Oracle. When the Oracle is used, the MC decides whether to write the new answer back into the AME for future reuse. Associative Memory Engine (AME). AME maintains an external memory that stores auxiliary knowledge as well as previously answered queries and their associated responses, and supports fast semantic retrieval (details in Appendix B ). Concretely, given a new query x_{t} at time t , AME retrieves a small set of K candidate memory entries \mathcal{M}_{t}=\{(x^{(i)},y^{(i)},m^{(i)})\}_{i=1}^{K}, where m^{(i)} denotes metadata (e.g., query category). The AME stores the question-answer pair for retrieval together with the qid (a unique identifier provided by the dataset), the category and the timestamp. In addition, AME supports write-back by storing newly generated high-quality entries (x_{t},y_{t},m_{t}) when instructed by the Meta Controller. Large-LLM Oracle. The oracle is a high-capability model that produces high-quality answers but incurs high inference cost. We use it as a fallback: the system queries the oracle when retrieval from AME is missing, ambiguous, or deemed unreliable by the Meta Controller. We denote the oracle’s response to query x_{t} as y_{t}^{\star}=\mathrm{Oracle}(x_{t}). Meta-Controller (MC). MC is a lightweight LLM that orchestrates the interaction among the user, AME, and the oracle. At each step, MC decides whether to answer using information retrieved from AME or to escalate to the oracle. When escalation occurs, MC also determines whether the newly generated query answer pair should be stored in AME for future reuse. Concretely, given a query x_{t} , MC first requests retrieval from AME and obtains \mathcal{M}_{t} . Conditioned on x_{t} and \mathcal{M}_{t} , MC either returns an answer directly, y_{t}=\mathrm{MC}(x_{t},\mathcal{M}_{t}) , or calls the oracle to obtain y_{t}^{\star}=\mathrm{Oracle}(x_{t}) and returns y_{t}=y_{t}^{\star} . In the latter case, MC may optionally write (x_{t},y_{t}^{\star},m_{t}) back into AME, where m_{t} denotes metadata inferred by MC.