When Should Dense Retrievers Be Updated in Evolving Corpora? Detecting Out-of-Distribution Corpora Using GradNormIR

Dayoon Ko, Jinyoung Kim, Sohyeon Kim, Jinhyuk Kim, Jaehoon Lee, Seonghak Song, Minyoung Lee, Gunhee Kim · 2025

Read When Should Dense Retrievers Be Updated in Evolving Corpora? Detecting Out-of-Distribution Corpora Using GradNormIR by Dayoon Ko, Jinyoung Kim, Sohyeon Kim, Jinhyuk Kim, Jaehoon Lee, Seonghak Song, Minyoung Lee, Gunhee Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01877.txt.

Introduction

Dayoon Ko and co-authors ask when a search engine's brain needs retraining as its library grows, in a 2025 paper. Dense retrievers turn text into number vectors and match questions to documents. New topics drift away from the training mix, and indexing them blindly can quietly wreck search quality.

Their tool, GradNormIR, sniffs out that drift before indexing, using gradient norms and no labeled data. Tests on the BEIR benchmark show the detector flags out-of-distribution corpora in time to trigger an update. Search quality then holds up as collections evolve. System teams gain an early-warning gauge that saves both failed searches and needless retraining runs.

Catalog record

CREATORDayoon Ko, Jinyoung Kim, Sohyeon Kim, Jinhyuk Kim, Jaehoon Lee, Seonghak Song, Minyoung Lee, Gunhee Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01877

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Should Dense Retrievers Be Updated in Evolving Corpora? Detecting Out-of-Distribution Corpora Using GradNormIR Dayoon Ko, Jinyoung Kim, Sohyeon Kim, Jinhyuk Kim, Jaehoon Lee, Seonghak Song, Minyoung Lee, Gunhee Kim Abstract Dense retrievers encode texts into embeddings to efficiently retrieve relevant documents from large databases in response to user queries. However, real-world corpora continually evolve, leading to a shift from the original training distribution of the retriever. Without timely updates or retraining, indexing newly emerging documents can degrade retrieval performance for future queries. Thus, identifying when a dense retriever requires an update is critical for maintaining robust retrieval systems. In this paper, we propose a novel task of predicting whether a corpus is out-of-distribution (OOD) relative to a dense retriever before indexing. Addressing this task allows us to proactively manage retriever updates, preventing potential retrieval failures. We introduce GradNormIR , an unsupervised approach that leverages gradient norms to detect OOD corpora effectively. Experiments on the BEIR benchmark demonstrate that GradNormIR enables timely updates of dense retrievers in evolving document collections, significantly enhancing retrieval robustness and efficiency. 1 Introduction With the exponential growth of digital content, information retrieval (IR) systems have become essential for delivering relevant information from massive document repositories ( Bajaj et al. 2016 ; Kwiatkowski et al. 2019 ) . Unlike traditional sparse retrieval methods ( Robertson et al. 2009 ; Ramos et al. 2003 ) that rely heavily on lexical overlap, dense retrievers ( Karpukhin et al. 2020 ; Izacard et al. 2022 ) utilize semantic embeddings to better understand query intent and retrieve documents with similar conceptual meanings, thus overcoming the constraints of exact term matching. Consequently, dense retrievers have gained significant attention in tasks demanding high semantic precision, such as question answering and personalized search. During training, these dense retrievers are optimized to enhance embedding similarity between queries and relevant passages while reducing similarity for irrelevant ones ( Karpukhin et al. 2020 ; Izacard et al. 2022 ) . Document embeddings are then precomputed and stored during indexing, enabling rapid retrieval during inference by identifying documents most similar to the test query embeddings. Figure 1: Motivation . In evolving corpora, indexing documents that dense retrievers fail to generalize to can severely degrade retrieval performance. Therefore, proactively detecting OOD corpora prior to indexing, without relying on available queries, is essential for maintaining retrieval effectiveness. To address this, we propose GradNormIR , an unsupervised method leveraging gradient norms to identify such OOD corpora. In the real world, corpora evolve rapidly due to technological advancements, societal changes, and emerging trends. This continuous evolution poses a substantial challenge for dense retrievers ( Chen et al. 2023 ) , which often struggle to generalize effectively to unseen documents in zero-shot scenarios ( Chen et al. 2022 ) . The problem is particularly critical in retrieval-augmented generation (RAG) systems ( Lewis et al. 2020 ) , where retriever performance directly impacts downstream tasks ( Petroni et al. ; Li et al. 2023a ; Ko et al. 2024 ; Kim et al. 2024 ) . For instance, consider the scenario illustrated in Figure. 1 : when a new corpus about Google’s quantum computing chip, Willow , is introduced, a dense retriever trained on existing content, such as the song Willow by Taylor Swift, may erroneously retrieve irrelevant documents. A query like "Tell me how the Willow works!" could mistakenly return information about the song Willow instead of the quantum chip. Anticipating when a retriever might fail due to such distributional shifts is crucial. Also, proactively identifying when to update the retriever ensures robustness and accuracy in dynamically evolving document streams. This challenge closely relates to the OOD generalization problem. Several approaches in information retrieval (IR) aim to improve a retriever’s performance on queries or documents significantly different from those encountered during training ( Izacard et al. 2022 ; Wang et al. 2021 ; Chen et al. 2022 ; Yu et al. 2022 ; Kasela et al. 2024 ; Besta et al. 2024 ; Chen et al. 2023 ) . One prominent approach utilizes a mixture-of-experts framework, employing a gating mechanism to select the most suitable expert retriever for each test query ( Kasela et al. 2024 ; Lee et al. 2024 ) . However, these methods depend heavily on predefined expert retrievers trained offline with explicit domain knowledge and established domain boundaries, making them less flexible for dynamically evolving corpora. Determining the appropriate timing to introduce new experts for emerging content remains challenging. To address this challenge, we introduce a novel practical task of predicting OOD corpora before indexing for a given dense retriever. Identifying OOD corpora in advance signals when retriever updates are needed. By detecting such corpora before indexing, we can proactively select a more suitable dense retriever or promptly update the current one, safeguarding inference-time performance. To achieve this, we propose GradNormIR , an unsupervised method designed to detect OOD documents within a corpus without relying on queries. Inspired by successful applications in image classification, where gradient norms effectively detect OOD images and estimate test-time accuracy without labeled data ( Huang et al. 2021 ; Xie et al. 2024 ) , GradNormIR leverages gradient norms from the contrastive loss to evaluate a retriever’s generalizability on a given corpus. Specifically, we introduce novel sampling strategies to effectively assign positive and negative instances for computing the contrastive loss. We assess our method on the BEIR benchmark, which encompasses multiple diverse datasets across various domains. Firstly, we demonstrate that GradNormIR effectively detects OOD documents likely to cause retrieval failures. Subsequently, we show that GradNormIR can select the most suitable retriever using only the corpus, without queries. Finally, we simulate evolving corpora using BEIR by sequentially introducing datasets following Ge et al. 2023 , and demonstrate how GradNormIR enables efficient retriever updates while maintaining performance. Our experiments validate both the importance of OOD detection for retrieval systems and GradNormIR’s efficacy in adapting to evolving corpora. In summary, our contributions are as follows: 1. We introduce a novel task of predicting OOD corpora before indexing, facilitating efficient and effective retriever updates in evolving corpora. 2. We propose GradNormIR, an unsupervised method that leverages gradient norms and novel sampling strategies to detect OOD documents and predict OOD corpora. 3. On the BEIR benchmark, we demonstrate both the necessity of the proposed task for a robust retrieval system and the effectiveness of GradNormIR via three practical use cases. 2 Related Work Information Retrieval. Recent advancements in text embeddings have significantly transformed the field of IR, particularly with the emergence of dense retrievers. The success of these models has primarily been driven by the availability of large-scale training datasets, including NQ ( Kwiatkowski et al. 2019 ) , MS-MARCO ( Bajaj et al. 2016 ) , HotpotQA ( Yang et al. 2018 ) , and NLI ( Gao et al. 2021 ) . A notable example is DPR ( Karpukhin et al. 2020 ) , which employs a dual-encoder architecture for open-domain question-answering, independently embedding queries and passages. Unsupervised approaches have also garnered attention for their ability to enhance the generalization of dense retrievers. Contriever ( Izacard et al. 2022 ) enlarges pre-training data using unsupervised data augmentation techniques for contrastive learning. Similarly, E5 ( Wang et

Read the complete text