Memory Bank Compression for Continual Adaptation of Large Language Models Conference: The 41st ACM/SIGAPP Symposium on Applied Computing; March 23–27, 2026; Thessaloniki, Greece The 41st ACM/SIGAPP Symposium on Applied Computing (SAC ’26), March 23–27, 2026, Thessaloniki, Greece DOI: 10.1145/3748522

Thomas Katraouras, Dimitrios Rafailidis · 2026

Read Memory Bank Compression for Continual Adaptation of Large Language Models Conference: The 41st ACM/SIGAPP Symposium on Applied Computing; March 23–27, 2026; Thessaloniki, Greece The 41st ACM/SIGAPP Symposium on Applied Computing (SAC ’26), March 23–27, 2026, Thessaloniki, Greece DOI: 10.1145/3748522 by Thomas Katraouras, Dimitrios Rafailidis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00756.txt.

Introduction

Thomas Katraouras and Dimitrios Rafailidis address how large language models can keep learning after deployment without forgetting what they know or drowning in stored data. Memory-augmented approaches give a model an external memory bank to consult, but in real data streams that bank grows without limit. Full fine-tuning, the alternative, is expensive and prone to wiping old knowledge.

Their model, MBC, compresses the memory bank through a codebook optimization strategy applied during online adaptation. Instead of keeping every stored item verbatim, the bank is distilled into a compact codebook that preserves what matters for future queries. The goal is continual adaptation that stays affordable as streams grow, updating a model with new information while retaining prior knowledge.

Catalog record

CREATORThomas Katraouras, Dimitrios Rafailidis
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-00756

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Memory Bank Compression for Continual Adaptation of Large Language Models Conference: The 41st ACM/SIGAPP Symposium on Applied Computing; March 23–27, 2026; Thessaloniki, Greece The 41st ACM/SIGAPP Symposium on Applied Computing (SAC ’26), March 23–27, 2026, Thessaloniki, Greece DOI: 10.1145/3748522.3779742 ISBN: 979-8-4007-2294-3/2026/03 CCS: Computing methodologies Natural language generation CCS: Computing methodologies Machine learning CCS: Computing methodologies Artificial intelligence CCS: Computing methodologies Natural language processing CCS: Computing methodologies Online learning settings Thomas Katraouras, Dimitrios Rafailidis Abstract . Large Language Models (LLMs) have become a mainstay for many everyday applications. However, as data evolve their knowledge quickly becomes outdated. Continual learning aims to update LLMs with new information without erasing previously acquired knowledge. Although methods such as full fine-tuning can incorporate new data, they are computationally expensive and prone to catastrophic forgetting, where prior knowledge is overwritten. Memory-augmented approaches address this by equipping LLMs with a memory bank, that is an external memory module which stores information for future use. However, these methods face a critical limitation, in particular, the memory bank constantly grows in the real-world scenario when large-scale data streams arrive. In this paper, we propose MBC, a model that compresses the memory bank through a codebook optimization strategy during online adaptation learning. To ensure stable learning, we also introduce an online resetting mechanism that prevents codebook collapse. In addition, we employ Key-Value Low-Rank Adaptation in the attention layers of the LLM, enabling efficient utilization of the compressed memory representations. Experiments with benchmark question–answering datasets demonstrate that MBC reduces the memory bank size to 0.3% when compared against the most competitive baseline, while maintaining high retention accuracy during online adaptation learning. Our code is publicly available at https://github.com/Thomkat/MBC. † † cc-license: by 1. Introduction Large Language Models (LLMs) 20 ; 33 have shown strong performance on a wide range of natural language processing tasks, including machine translation 39 , summarization 35 , question answering 31 , and advanced reasoning 44 . They are now widely employed in many applications such as search engines 47 and personal assistants 4 . However, a major limitation of these models is that they are static 10 . Once trained, their parameters reflect only the data seen during training, and they cannot easily incorporate new knowledge. This leads to the problem of knowledge cutoff, where the model’s internal knowledge becomes outdated as new information appears 20 ; 33 . To address this limitation, Retrieval-Based Augmentation (RAG) strategies have been introduced 14 ; 40 . A frozen LLM uses a retriever to fetch relevant passages from an external corpus at inference time, providing the model with access to up-to-date information without retraining 14 . However, RAG methods face several challenges. They depend on nearest-neighbor search, which adds computational overhead and latency 40 . The quality of the retrieval affects the LLM performance, and errors in retrieval propagate directly to the generator 46 . Retrievers also often require domain-specific tuning and may struggle to generalize across domains 19 . Furthermore, the retrieved passages are typically concatenated with the query in the input context window, limiting the model’s ability to fully utilize the information and creating issues when the combined length exceeds the model’s capacity 14 . These issues limit the scalability of RAG for long-term adaptation in streaming environments, reflecting on the real-world scenario. To solve the problem of LLMs’ long-term adaptation, continual learning methods have been proposed 30 ; 43 . In this setting, models are updated as new data arrive. The simplest approach, full fine-tuning, also known as uniform fine-tuning 9 since all tokens are weighted equally, updates all parameters on the new data. While effective for small models, this is computationally expensive for large LLMs and is prone to catastrophic forgetting 18 , where performance on previously learned knowledge degrades as the model is optimized on new information. Parameter-efficient fine-tuning (PEFT) methods 41 such as adapters 7 ; 22 , prefix-tuning 15 , and LoRA 8 address the computational cost by introducing small trainable modules while keeping most parameters frozen. These approaches reduce the training overhead, however, they still require gradient-based updates at deployment time, which is impractical in streaming scenarios. Additional strategies have been proposed to make updates selective and stable, for example by restricting updates to predefined salient spans 5 , by meta-learning token importance weights 9 , or by interleaving past and new examples through replay 2 ; 28 . Despite these refinements, the fundamental limitations persist, namely the need for repeated optimization, substantial computational and latency overhead, and continued vulnerability to catastrophic forgetting 30 . A promising direction for continual learning of LLMs is memory augmentation 6 ; 21 ; 38 . Instead of retrieving raw text from an external corpus, information is stored directly in a structured memory module, which can also be updated dynamically. At inference time, the model can draw on these stored representations to adapt its behavior and incorporate new information. This avoids repeated gradient updates and provides a direct connection between the stored knowledge and model’s computations 6 . However, memory augmentation introduces challenges as more documents are processed, the memory bank constantly grows. This increases storage costs and slows down inference, since the model must attend to an ever-growing set of contexts 45 . Expanding memory without disrupting the model’s original behavior remains difficult, and as a consequence, the memory augmentation strategies require retraining or fine-tuning to remain effective. More recently, to overcome this shortcoming, memory-augmented frameworks have been proposed that store learned modulation parameters for each document in an external memory bank 32 . These approaches keep the base model frozen, condition the model on the entire memory bank rather than a single retrieved document, and avoid further fine-tuning during adaptation, while mitigating catastrophic forgetting. Nevertheless, in the real-world scenario where the document stream reaches hundreds of thousands or millions of entries, the memory bank grows very large and becomes difficult to manage. This highlights scalability as an open problem in memory-augmented systems, alongside the need to balance adaptation, efficiency, and stability. In this paper, we propose MBC, a model that compresses the memory bank while maintaining high performance on downstream question-and-answer (QA) tasks. Specifically, we make the following contributions: • We propose a memory bank compression method based on a codebook optimization strategy, which stores indices to this codebook instead of full document representations. In addition, we introduce an online resetting strategy to prevent codebook collapse and ensure balanced code utilization and stable training. • We employ Key-Value Low-Rank Adaptation targeted only to the attention layers of the model. In doing so, we improve the proposed model’s ability to adapt when new data arrive without requiring full fine-tuning. We conduct experiments on benchmark QA datasets, comparing our MBC model with baseline methods. Our results demonstrate that MBC significantly reduces the memory bank size to 0.3%, when compared with the initial size of baseline strategies, and improves the QA accuracy. Furthermore, our model maintains a high retention accuracy when evaluated for catastrophic forgetting during the challenging online adaptation scenario. The remainder of the paper is structured as follows: Section 2 formulates the problem of online learning in LLMs. Section 3 outlines the MBC model and Section 4 provides the experimental evaluation. Finally, Section 5

Read the complete text