On Correlating Factors for Domain Adaptation Performance

Göksenin Yüksel, Jaap Kamps · 2025

Read On Correlating Factors for Domain Adaptation Performance by Göksenin Yüksel, Jaap Kamps (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14466.txt.

Introduction

Yüksel and Kamps studied what makes domain adaptation work for dense retrievers in twenty twenty-five. Dense retrievers are search systems that match queries to documents by meaning. They perform well in their home domain but falter when topics shift, a common problem in zero-shot search.

One fix is generating practice queries in the new domain. The authors test which factors predict success, using similarity measures between generated queries and target domains. A case study compares two strong adaptation methods. They find the type mix of generated queries matters a great deal. Queries that resemble the target domain lift adaptation performance. The study stresses tailoring generated queries to the domain at hand rather than using generic ones.

Catalog record

CREATORGöksenin Yüksel, Jaap Kamps
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-14466

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On Correlating Factors for Domain Adaptation Performance Göksenin Yüksel, Jaap Kamps Abstract Dense retrievers have demonstrated significant potential for neural information retrieval; however, they lack robustness to domain shifts, limiting their efficacy in zero-shot settings across diverse domains. In this paper, we set out to analyze the possible factors that lead to successful domain adaptation of dense retrievers. We include domain similarity proxies between generated queries to test and source domains. Furthermore, we conduct a case study comparing two powerful domain adaptation techniques. We find that generated query type distribution is an important factor, and generating queries that share a similar domain to the test documents improves the performance of domain adaptation methods. This study further emphasizes the importance of domain-tailored generated queries. 1 Introduction Dense retrieval (DR) methods are extremely fast and easy to index ( Zhao et al. 2023 ) . However, these models have to solve the much more challenging task of mapping inputs independently to a meaningful vector space ( Thakur et al. 2021a ) . Compared to other neural IR methods, they show poor zero-shot performance across novel domains ( Thakur et al. 2021b ) . Hence, there is a need to better understand the conditions and factors that contribute to the (lack of) zero-shot performance. Ren et al. 2023 analyze various attributes affecting the zero-shot performance of the DR methods. They provide an in-depth empirical analysis of different factors on zero-shot retrieval performance In this paper, we set out to analyze the variables that may contribute to a successful domain adaptation. We examine the possible correlating factors for successful domain adaptation from the standpoint of ( Ren et al. 2023 ) . We choose to carry out our analysis on the GPL ( Wang et al. 2022 ) method, as this method was the most successful domain adaptation method for dense retrievers without taking hybrid models into account ( Ren et al. 2023 ) . We further conduct a case study using the InPars ( Bonifacio et al. 2022 ) method and test the applicability of our findings in cross-encoder domain adaptation. We extend the factors of ( Ren et al. 2023 ) with various attributes regarding the generated queries, such as: generated query vocabulary overlap between source/test queries, generated query vocabulary overlap between source/test documents, generated query type distribution entropy, and generated query type distribution overlap between source/test queries. Our main contributions are the following. First, we find that the GPL method transfers to LoTTE datasets and shows substantial improvements in NDCG@10 compared to the baseline. This indicates that the GPL method is a robust domain adaptation method that shows performance improvements over a broad spectrum of domains. Second, we discover that "Generated Query Type Entropy is one of the critical attributes that correlates positively with the performance of the domain-adapted retriever. Third, we further find that the overlap of the generated domain to the test domain is an indicator of domain adaptation performance regardless of the framework. The rest of this paper continues with related work (§ 2 ), our approach and experimental setup (§ 3 ), the corresponding results and analysis (§ 4 ), and ends with discussion and conclusions (§ 5 ). 2 Related Research This section discusses earlier research on performance factors and on domain adaptation. Factors for zero shot performance To the best of our knowledge, Ren et al. 2023 is the first paper to analyze attributes affecting the zero-shot performance of dense retriever methods. They provide an in-depth empirical analysis by investigating the effect of different factors on zero-shot retrieval performance, such as query/passage vocabulary overlap, query-type distribution, query scale, and lexical bias. Domain Adaptation Query generation (QG) models generate synthetic queries from documents. They introduce synthetic training data by generating new queries to form a positive pair with the document from which they have been generated. This procedure introduces domain-specific synthetic data to fine-tune the model. QGen Ma et al. 2021 uses a transformer to create synthetic queries for the training data with binary relevance labels. GPL Wang et al. 2022 extends the work of QGen by using the cross-encoder model to generate pseudo labels for synthesized query-document pairs. GPL utilizes QG and successfully adapts the dense-retriever to the domain by showing improvement on 18 of 19 BEIR datasets Wang et al. 2022 . Moreover, InPars Bonifacio et al. 2022 uses a large language model (LLM) to generate synthetic queries. They use OpenAI’s public API to prompt GPT-3 to generate plausible queries for a document, showcasing that fine-tuning cross-encoders in this synthetic dataset yield state-of-the-art results in many datasets. Their research further reiterates that the generated synthetic queries provide a solid learning signal to fine-tune neural IR models. 3 Method This section details our experimental setup. Vocabulary Overlap We consider each vocabulary’s top 10K most frequent words (excluding stopwords). We calculate the percentage of vocabulary overlap by the weighted Jaccard Ioffe 2010 for each pair of query and document sets. Query Type Distribution Following Ren et al. 2023 , we first categorize each query and generate the query to a specific type depending on the keywords. The used keywords are in Appendix A. Later, we measure the entropy of the query type distribution. Higher entropy yields a more uniform distribution and, thus, more diverse query types. Furthermore, we use cross-entropy to analyze the overlap of query-type distributions. Additional Factors We analyze the generated queries to understand their similarities to test and source domains. For this, we utilize generated query-source query, generated query-test query, generated query-source document, and generated query type distribution overlap between source/test queries. We analyze the Spearman correlation for both models’ improvement and retention capabilities with respect to the factors. 1 1 1 For the CQADupstack dataset, we take the average for all the factors Models For BEIR tasks, we use the open source models and generated queries from Wang et al. 2022 . Moreover, we use the pipeline described in Wang et al. 2022 to adapt the GPL/msmarco-distilbert-margin-mse model on the LoTTE dataset. Furthermore, we use the generated queries provided by ( Bonifacio et al. 2022 ) . BEIR ( Thakur et al. 2021b ) combines several existing datasets into a heterogeneous suite for “zero-shot IR” tasks spanning bio-medical, financial, and scientific domains. Our paper utilizes open-access datasets from BEIR, which has 14 out of 19 collections. LoTTE ( Santhanam et al. 2022 ) is a new dataset for Long-Tail Topic-stratified Evaluation for IR. To complement the out-of-domain tests of BEIR, LoTTE focuses on natural user queries that pertain to long-tail topics, ones that an entity-centric knowledge base like Wikipedia might not cover. LoTTE consists of 12 test sets, each with 500–2,000 queries and 100k–2M passages . ((a)) Factors affecting domain adaptation performance: GPL on different collections. ((b)) Factors affecting domain adaptation performance: GPL’s T5 vs InPars query generation. Figure 1: Comparison of factors affecting domain adaptation performance measured in NDCG@10 improvement. 4 Results This section discusses our results on the factors for successful domain adaptation. Corpus BM25 MSMARCO GPL LoTTE Search Test Queries Writing 63.2 70.6 \textbf{77.1}^{{+6.5}} Recreation 59.8 62.8 \textbf{71.0}^{{+8.2}} Science 38.6 46.4 \textbf{52.0}^{{+5.2}} Technology 44.5 57.6 \textbf{64.1}^{{+6.5}} Lifestyle 68.1 77.0 \textbf{82.8}^{{+5.8}} Pooled 52.4 62.1 \textbf{65.2}^{{+3.1}} LoTTE Forum Test Queries Writing

Read the complete text