A Parametric Memory Head for Continual Generative Retrieval Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809725 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Retrieval models and ranking CCS: Information systems Learning to rank CCS: Information systems Language models
Kidist Amde Mekonnen, Yubao Tang, Maarten de Rijke · 2026
Read A Parametric Memory Head for Continual Generative Retrieval Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809725 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Retrieval models and ranking CCS: Information systems Learning to rank CCS: Information systems Language models by Kidist Amde Mekonnen, Yubao Tang, Maarten de Rijke (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23388.txt.
Introduction
Mekonnen, Tang, and de Rijke tackle a weakness of generative retrieval, where one neural model stores the whole index in its weights. Retraining on fresh documents lifts new results while damaging earlier slices, so the model forgets what it once knew. Their fix adds a small add-on memory that steadies the model without touching its core weights. Tests on two retrieval collections show earlier slices are retained far better at little cost to new documents.
Catalog record
| CREATOR | Kidist Amde Mekonnen, Yubao Tang, Maarten de Rijke |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60366 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.23388
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Parametric Memory Head for Continual Generative Retrieval Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia DOI: 10.1145/3805712.3809725 ISBN: 979-8-4007-2599-9/2026/07 CCS: Information systems Retrieval models and ranking CCS: Information systems Learning to rank CCS: Information systems Language models Kidist Amde Mekonnen, Yubao Tang, Maarten de Rijke Abstract . Generative information retrieval (GenIR) consolidates retrieval into a single neural model that decodes document identifiers (docids) directly from queries. While this model-as-index paradigm offers architectural simplicity, it is poorly suited to dynamic document collections. Unlike modular systems, where indexes are easily updated, GenIR’s knowledge is parametrically encoded in its weights; consequently, standard adaptation methods such as full and parameter-efficient fine-tuning can induce catastrophic forgetting. We show that sequential adaptation improves retrieval on newly added documents but substantially degrades performance on earlier slices, exposing a pronounced stability–plasticity trade-off. To address this, we propose post-adaptation memory tuning (PAMT), a memory-only stabilization stage that augments an adapted model with a modular parametric memory head (PMH). PAMT freezes the backbone and attaches a product-key memory with fixed addressing. During prefix-trie constrained decoding, decoder hidden states sparsely query PMH to produce residual corrections in hidden space; these corrections are mapped to score adjustments via the frozen output embedding matrix, computed only over trie-valid tokens. This guides docid generation while keeping routing and backbone parameters fixed. To limit cross-slice interference, PAMT updates only a fixed budget of memory values selected using decoding-time access statistics, prioritizing entries frequently activated by the current slice and rarely used in prior sessions. Experiments on MS MARCO and Natural Questions under sequential, disjoint corpus increments show that PAMT substantially improves retention on earlier slices with minimal impact on retrieval performance for newly added documents, while modifying only a sparse subset of memory values per session. † † cc-license: by 1. Introduction Generative information retrieval (GenIR) reformulates document retrieval as an autoregressive generation task, mapping natural language queries directly to unique document identifiers (docids) ( 32 ; 46 ; 45 ) . By internalizing corpus knowledge in model parameters, GenIR enables end-to-end optimization, substantial index compression, and reuse of semantic priors from pretrained language models ( 53 ; 49 ; 48 ; 43 ; 31 ) . Training typically uses an encoder–decoder architecture with two complementary objectives: indexing , which associates document content with docids, and retrieval , which maps queries to docids. At inference time, GenIR employs constrained decoding (e.g., prefix tries) to restrict generation to valid identifiers. Challenge: continual retrieval over dynamic corpora. Real-world collections are dynamic: new documents arrive continuously, and retrieval systems must integrate them without compromising performance on previously indexed data ( 30 ) . This is particularly challenging for GenIR. Unlike modular IR pipelines with explicit index structures, GenIR entangles query–docid mappings in model parameters, so updating the model to accommodate new documents can disrupt established retrieval behavior. Joint retraining (or rehearsal) on old and new data is the most reliable way to preserve performance, but it is often prohibitively expensive at scale. In contrast, adapting only on newly arrived documents is efficient, but it typically causes catastrophic forgetting ( 29 ; 7 ) , leading to a stability–plasticity trade-off: gains on the new slice ( plasticity ) often come at the cost of degraded retrieval on earlier content ( stability ). Existing approaches. Prior work on continual GenIR mitigates forgetting during adaptation , through replay or synthetic rehearsal, identifier design and index-side strategies, or parameter-efficient updates such as prompt tuning and adapters. While effective, these methods typically couple plasticity and retention in a single objective, making performance under long sequences of continual updates sensitive to gradient trade-offs and sampling ratios (see Section 2 ). Empirical motivation. We analyze continual adaptation in GenIR by training on an initial corpus and sequentially incorporating disjoint document batches. We consider two representative docid designs: (i) semantic numeric identifiers such as product-quantized codes ( 53 ; 5 ; 52 ; 51 ; 44 ) , and (ii) keyword-based identifiers such as URLs or titles ( 34 ; 21 ; 3 ; 39 ; 31 ) . We find that parametric adaptation methods, including full fine-tuning and parameter-efficient updates such as LoRA ( 10 ) , reliably improve retrieval on newly added documents but substantially degrade performance on earlier slices. In contrast, when model parameters are frozen, expanding the constrained-decoding candidate set alone causes only modest retention drift, while zero-shot transfer to future-slice identifiers remains limited. These controls indicate that the dominant source of degradation is update-induced interference in the learned query–docid mapping rather than search-space growth alone. They also indicate that adapting to each new slice remains necessary for plasticity, even though it harms stability. This motivates a post-adaptation mechanism that improves retention after learning new slices, without re-optimizing the full backbone. Our approach. We propose post-adaptation memory tuning (PAMT), a two-stage framework that decouples learning new docid mappings from post-adaptation decoder-side calibration. In Stage 1, the GenIR backbone is adapted to the newly arrived slice using a standard adaptation method. In Stage 2, the adapted backbone is frozen and augmented with a parametric memory head (PMH), implemented as a product-key memory ( 20 ) . During prefix-trie constrained decoding, PMH produces sparse hidden-space corrections whose effect on next-token scores is computed only over trie-valid tokens. To limit cross-slice interference, Stage 2 updates only a fixed budget of memory values selected using decoding-time access statistics, while keeping the backbone and PMH routing components fixed during the stabilization stage. This yields a value-only post-adaptation calibration step targeted at the docid-decoding interface, rather than further backbone optimization. Importantly, PAMT does not attempt to restore pre-adaptation routing; instead, it performs calibration under the routing pattern induced by Stage 1. Contributions. (i) We provide an empirical characterization of catastrophic forgetting in continual GenIR, quantifying the stability–plasticity trade-off across semantic numeric and keyword-based docid schemes under full and parameter-efficient adaptation. (ii) We introduce post-adaptation memory tuning (PAMT), an adapt-then-stabilize framework that improves retention after session-level adaptation through a memory-only stabilization stage, without replaying legacy supervision. (iii) We introduce a decoder-side parametric memory head (PMH) for continual GenIR that enables sparse post-adaptation value-only calibration under prefix-trie constrained decoding, allowing retention to be improved without further backbone updates or routing changes during Stage 2. 2. Related Work Continual learning. Catastrophic forgetting ( 29 ; 7 ) remains a central challenge when training neural models on non-stationary data. Common mitigation strategies include experience replay ( 36 ; 37 ; 4 ) , parameter regularization such as EWC ( 17 ) , and knowledge distillation ( 22 ) . In GenIR, these strategies face additional constraints: replay requires storing queries and/or documents, raising privacy concerns ( 25 ) and scaling poorly over many slices ( 6 ; 38 ; 13 ) ; regularization becomes harder to tune as the docid space expands; and distillation introduces additional objectives and hyperparameters that may be sensitive to slice-level distribution shift. Continual learning in GenIR. Prior work adapts GenIR models to evolving corpora primarily through training-time updates. Replay-based