Impact of Shallow vs. Deep Relevance Judgments on BERT-based Reranking Models Conference: Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR); July 18, 2025; Padua, Italy Proceedings of the 2025 International ACM SIGIR Confe
Gabriel Iturra-Bocaz, Danny Vo, Petra Galuščáková · 2025
Read Impact of Shallow vs. Deep Relevance Judgments on BERT-based Reranking Models Conference: Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR); July 18, 2025; Padua, Italy Proceedings of the 2025 International ACM SIGIR Confe by Gabriel Iturra-Bocaz, Danny Vo, Petra Galuščáková (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-23191.txt.
Introduction
Gabriel Iturra-Bocaz, Danny Vo, and Petra Galuščáková study, in twenty twenty-five, how the depth of relevance judgments affects search ranking models. Search systems learn from judged data sets. Shallow sets have many queries but few judgments per query. Deep sets have fewer queries but many judgments each. The team trained BERT-based rerankers, models that reorder search results, on both types. Tests ran on two known collections, MS MARCO and LongEval.
Shallow-judged data generally gave better generalization and effectiveness. The broader range of contexts likely helps. Deep-judged data lagged, though the gap could be softened by adding more negative training examples, results the model must learn to rank low.
Catalog record
| CREATOR | Gabriel Iturra-Bocaz, Danny Vo, Petra Galuščáková |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-23191
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Impact of Shallow vs. Deep Relevance Judgments on BERT-based Reranking Models Conference: Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR); July 18, 2025; Padua, Italy Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR) (ICTIR ’25), July 18, 2025, Padua, Italy DOI: 10.1145/3731120.3744602 ISBN: 979-8-4007-1861-8/2025/07 CCS: Information systems Retrieval models and ranking CCS: Information systems Evaluation of retrieval results Gabriel Iturra-Bocaz, Danny Vo, Petra Galuščáková Abstract . This paper investigates the impact of shallow versus deep relevance judgments on the performance of BERT-based reranking models in neural Information Retrieval. Shallow-judged datasets, characterized by numerous queries each with few relevance judgments, and deep-judged datasets, involving fewer queries with extensive relevance judgments, are compared. The research assesses how these datasets affect the performance of BERT-based reranking models trained on them. The experiments are run on the MS MARCO and LongEval collections. Results indicate that shallow-judged datasets generally enhance generalization and effectiveness of reranking models due to a broader range of available contexts. The disadvantage of the deep-judged datasets might be mitigated by a larger number of negative training examples. 1. Introduction In neural Information Retrieval (IR), the method of collecting training data has a significant impact on the performance of retrieval models trained on this data. The quality and structure of the training data influence how well the models learn to distinguish relevant from non-relevant documents during the inference. Data collection approaches in IR (either for training or evaluation) which use documents manually judged for their relevance to the queries, can be generally categorized into two primary sets: deep and shallow . Deep-judged datasets contain fewer queries, each with numerous relevance judgments. These datasets, often created through pooling in IR evaluations, typically consist of tens of queries, each associated with hundreds of relevance judgments. In contrast, shallow-judged datasets use many queries, each with few relevance judgments, offering broader coverage but less depth per query. Shallow-judged datasets are often derived from real-world queries in production systems and are often used for both training and evaluation. For simplicity, we refer to deep-judged datasets as deep datasets and shallow-judged datasets as shallow datasets throughout this paper. The same abbreviated terminology applies to judgments (deep judgments and shallow judgments) collected on deep and shallow datasets. Shallow and deep datasets were introduced by Yilmaz and Robertson 2009 , who explored the trade-off between the number of queries and the number of judgments per query for training learning-to-rank systems. In this paper, we re-examine the effectiveness of shallow versus deep judgments, for training a BERT-based reranker ( Devlin et al. 2019 ) . Rather than relying solely on the statistical pattern learning used in traditional learning-to-rank, we use BERT-based rerankers which capture semantic relationships in query-document pairs during the training phase. We additionally study the role of negative sampling. The central research question of this paper is: what is the impact of training rerankers on shallow datasets versus deep datasets? While shallow judgments are expected to have an advantage due to the larger number of diverse contexts, we aim to determine if, and under what circumstances, deep judgments can outperform shallow ones. We also focus on the effect of the negative relevance judgments, as they are typically much easier to acquire than the true positive judgments by using automatized methods. Since creating relevance judgments is time-consuming and resource-intensive, determining whether manual efforts in building evaluation datasets can be effectively reused for training could have a significant impact on the IR community. We conduct our comparison using two MS MARCO collections and a LongEval collection, all of which contain shallow and deep judgments 1 1 1 Our code is available at https://github.com/iai-group/ictir2025-relevance . In the next section, we review related work, focusing on the impact of training data on reranking model performance. Section 3 describes the collections used in the experiments, while Section 4 outlines the experiments and analyzes the results. We discuss the results in Section 5 . Finally, Section 6 concludes the paper. 2. Related Work The introduction of BERT by Devlin et al. 2019 marked a significant milestone in IR, allowing models to capture complex semantic relationships within text ( Yates et al. 2021 ) . This advancement paved the way for BERT’s application in reranking, where it refines results obtained from traditional retrieval methods like BM25. The effectiveness of BERT as a reranker was first demonstrated by Nogueira et al. 2019 . They introduced two reranking setups, MonoBERT and DuoBERT, which transformed the ranking task into pointwise and pairwise classifications, respectively. In our experiments, we adopt the MonoBERT setup. Nogueira et al. ( Nogueira et al. 2019 ; Nogueira and Cho 2019 ; Nogueira et al. 2020 ) further explored the dependence of the MonoBERT reranker on the number of training instances (query / document pairs). They found that approximately 10,000 training instances are required to outperform the baseline BM25 model on the MS MARCO collection. Like us, they fine-tuned the vanilla BERT model. Other experiments, such as those by Zhang et al. 2020 , show that BERT pre-trained on the full MS MARCO collection can outperform BM25, even on out-of-domain data. However, fine-tuning this pre-trained BERT on too small in-domain dataset can degrade performance. Althammer et al. 2023 explored active learning strategies for the effective selection of training data for neural models. Although they observed a significant effect of training data selection on model performance, they did not find any active learning strategy that outperformed random selection of training instances. Additionally, the selection of negative samples also proved to have a significant effect on retrieval quality ( Cai et al. 2022 ; Qu et al. 2021 ; Xiong et al. 2021 ) . Our work is mostly related to the work of Yilmaz and Robertson 2009 , who studies the differences between deep and shallow relevance judgments in learning-to-rank systems on which they show that the shallow judgments are more cost effective than deep judgments. Shallow and deep judgments were also studied when used in the evaluation. Zobel 1998 examined the optimal pool depth and claimed that the pool depth of 100 documents might be enough for acquiring reliable results. Lu et al. 2017 explored to which extent can deep judgment-based methods be effective for evaluating shallow-based judgments. They propose novel methods for such evaluation, based on a prediction of a gain of unjudged documents. 3. Datasets We utilize both shallow and deep judgments from the MS MARCO and LongEval collections. The use of the terms deep and shallow here refers to how training data is gathered. Specifically, to the trade-off between query coverage and labeling approach between the number of queries vs the number of relevance judgment per query. In this context, a deep dataset is constructed by collecting many relevance judgments for each individual query, resulting in thorough labeling for a small set of queries. This approach provides insight into the relevance judgment landscape for each query but limits the diversity of queries represented in a training set. On other hand, a shallow dataset is built by collecting only a few relevance judgments per query, but considering a much broader set of queries, which increases