Remining Hard Negatives for Generative Pseudo Labeled Domain Adaptation

Goksenin Yuksel, David Rau, Jaap Kamps · 2025

Read Remining Hard Negatives for Generative Pseudo Labeled Domain Adaptation by Goksenin Yuksel, David Rau, Jaap Kamps (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14434.txt.

Introduction

Search engines that learn from fake queries can still stumble on hard example documents. Goksenin Yuksel and co-authors improve a leading adaptation method in this 2025 paper. The method teaches a compact retriever using synthetic queries and tricky negative examples mined once at the start.

The team noticed the adapted retriever already surfaces negatives closer to the mark than the starter did. So they refresh the negative pool mid-training, mining better hard cases from the improving model itself. The refreshed method, R-GPL, lifted ranking scores on thirteen of fourteen test collections in one benchmark and nine of twelve in another. The gain costs little: the same training loop, with one extra mining pass.

Catalog record

CREATORGoksenin Yuksel, David Rau, Jaap Kamps
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-14434

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Remining Hard Negatives for Generative Pseudo Labeled Domain Adaptation Goksenin Yuksel, David Rau, Jaap Kamps Abstract Dense retrievers have demonstrated significant potential for neural information retrieval; however, they exhibit a lack of robustness to domain shifts, thereby limiting their efficacy in zero-shot settings across diverse domains. A state-of-the-art domain adaptation technique is Generative Pseudo Labeling (GPL). GPL uses synthetic query generation and initially mined hard negatives to distill knowledge from cross-encoder to dense retrievers in the target domain. In this paper, we analyze the documents retrieved by the domain-adapted model and discover that these are more relevant to the target queries than those of the non-domain-adapted model. We then propose refreshing the hard-negative index during the knowledge distillation phase to mine better hard negatives. Our remining R-GPL approach boosts ranking performance in 13/14 BEIR datasets and 9/12 LoTTe datasets. Our contributions are (i) analyzing hard negatives returned by domain-adapted and non-domain-adapted models and (ii) applying the GPL training with and without hard-negative re-mining in LoTTE and BEIR datasets. 1 Introduction Dense passage retrieval Zhao et al. (2023) is a key approach for text retrieval on large corpora, in the crucial first stage of modern NLP pipelines. The underlying idea is to represent queries and passages as low-dimensional vectors (also known as embeddings) to measure relevance by comparing the similarity between the query and document embeddings Thakur et al. (2021a) . Embedding queries and passages independently make these models efficient and applicable as first-stage retrievers on large corpora. Document embeddings can be pre-computed and stored, reducing computational overhead at inference time. However, they suffer from overfitting to domain-specific training data Ma et al. (2021) and underperforming in novel test domains Thakur et al. (2021b) . More robust approaches, such as cross-encoder and late interaction models, exist in the literature but come with computational and memory overheads. Therefore, they are limited in inference time efficiency, which is the prime concern for any practical application in production systems. Figure 1: Score distribution of top 100 documents retrieved before and after domain adaptation. Estimated relevance scores based on the teacher model from GPL. The lower effectiveness of dense retrieval models out-of-domain hinders the application of dense retrieval systems in zero-shot settings, making them less viable choices compared to cross-encoders or late-interaction models. To overcome this limitation, “domain adaptation” methodologies have been proposed. Figure 1 shows the shift in score distribution between the base zero-shot model and the domain-adapted model. Documents retrieved by the domain-adapted model are clearly more relevant to the query than those retrieved by the non-domain-adapted model. This demonstrates the potential of domain adaption of models, but also prompts the question if we can exploit this to further improve the domain adaptation. This directly motivates the main research question of this paper: Can the domain-adapted model provide more informative hard negatives to further adapt the model to the target domain? Unsupervised domain adaptation exploits two methods that improve performance for adapting models out-of-domain: knowledge distillation (KD) and query generation (QG) Wang et al. (2022) ; Ma et al. (2021) ; Thakur et al. (2021b) . Considering that human-generated relevance judgment is quite costly Xin et al. (2022) ; Thakur et al. (2021a) ; Ma et al. (2021) ; Bonifacio et al. (2022) , robust models can be used effectively for pseudo labeling instead, and KD becomes an essential approach to improving the capacity of dense retrievers Thakur et al. (2021a) ; Hofstätter et al. (2020) . Given a source domain with sufficient training signals, the goal is to transfer the dense retrieval (DR) model to a target domain without access to human editorial relevance labels. Query generation (QG) models introduce synthetic training data by generating new queries to form positive pairs with documents ( Balog et al., 2006 , e.g.,) . Combining both procedures introduces domain-specific synthetic data to further fine-tune models in a specific domain without the need for human labels. GPL ( Wang et al., 2022 ) has demonstrated to be a very effective non-hybrid dense retriever domain adaptation method Ren et al. (2023) . GPL distills knowledge from the cross-encoder model to the dense retrieval model using synthetic training data. To this end, GPL performs hard-negative mining step at the beginning of training, using pre-trained MSMARCO dense retrievers. The hard negatives are mined before the training and remain unchanged throughout the entire domain-adaptation process. GPL shows significant performance improvements on 18 of 19 BEIR datasets over previous methods. In this paper, we extend GPL by proposing to remine hard negatives during the domain adaptation process. The main idea is that instead of using the ranker that has been trained on the source domain to mine hard negatives, we employ the model undergoing the domain adaptation. During training, the model learns to retrieve more relevant hard negatives, optimizing the margin of the top-ranked documents. Our experiments show that remining hard negatives by the domain-adapted model during training can improve performance over using initially retrieved, static hard negatives. Our analysis shows that the hard negatives retrieved by the domain-adapted model are significantly more relevant to the queries than pre-retrieved hard negatives. Our main contributions are the following. First, we complement the original work of GPL by evaluating it on the LoTTE benchmark. Second, we improve GPL by proposing to continuously remine hard negatives with the model undergoing domain adaptation and demonstrate performance improvements on BEIR and LoTTe benchmarks. Our method boosts the ranking performance of the domain-adapted model in 13 out of 14 datasets on BEIR and in 9 out of 12 on LoTTE. Third, in an extensive analysis we investigate hard negatives returned by the domain-adapted retriever and show why remining hard negatives is effective for domain adaptation. The rest of this paper is structured as follows. § 2 embeds our work in the literature. § 3 details our novel domain adaptation approach based on remining hard negatives. § 4 details our experimental setup. § 5 shows the experimental results demonstrating the retrieval effectiveness of domain adaptation. § 6 conducts an extensive analysis how remining hard negatives impacts domain adaptation. The paper ends in § 7 with discussion and conclusions. 2 Related Work This section details related work on robust ranking models, on unsupervised domain adaptation, and on hard negative mining. 2.1 Robust Ranking Models Cross-encoders Nogueira and Cho (2020) demonstrate better performance and are less prone to the out-of-domain problem Thakur et al. (2021b) . They concatenate the query and passage and predict a relevance score using cross-attention. Despite their superior performance and resilience to domain shift, the requirement to concatenate the query and passage renders them less suitable for a production setting due to extremely high computational overhead. Similarly, late interaction models have shown robustness to domain shift while delivering performance comparable to cross-encoders Thakur et al. (2021b) ; Khattab and Zaharia (2020) ; Santhanam et al. (2022) . These models delay the interaction between query and document tokens until the final stage. Consequently, they provide a significant speedup compared to cross-encoder models that use cross-attention from the beginning Khattab and Zaharia (2020) . However, late interaction models are still inefficient compared to dense retrieval models due to the need to compare every query and document token embedded

Read the complete text