Disentangling Locality and Entropy in Ranking Distillation

Andrew Parry, Debasis Ganguly, Sean MacAvaney · 2025

Read Disentangling Locality and Entropy in Ranking Distillation by Andrew Parry, Debasis Ganguly, Sean MacAvaney (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21058.txt.

Introduction

Search engines learn ranking from big teacher models, but which part of the training recipe earns the gains was muddy. Andrew Parry and two co-authors pull the recipe apart in 2025, separating how training examples are chosen from how their labels are written. Popular pipelines had grown baroque, stacking a dozen models with murky data origins.

Theory shows the two choices bend the model along independent axes, and augmentation can fix a biased ranker under stated conditions. Experiments deliver a surprise: hard example mining, a hero of older training styles, can be useless or harmful under distillation. The entropy of the teacher's rankings steers learning more than the sampling does. The authors urge simpler, cheaper recipes aimed at the real levers.

Catalog record

CREATORAndrew Parry, Debasis Ganguly, Sean MacAvaney
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21058

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Disentangling Locality and Entropy in Ranking Distillation Andrew Parry, Debasis Ganguly, Sean MacAvaney Abstract The training process of ranking models involves two key data selection decisions: a sampling strategy (which selects the data to train on), and a labeling strategy (which provides the supervision signal over the sampled data). Modern ranking systems, especially those for performing semantic search, typically use a “hard negative” sampling strategy to identify challenging items using heuristics and a distillation labeling strategy to transfer ranking “knowledge” from a more capable model. In practice, these approaches have grown increasingly expensive and complex—for instance, popular pretrained rankers from SentenceTransformers involve 12 models in an ensemble with data provenance hampering reproducibility. Despite their complexity, modern sampling and labeling strategies have not been fully ablated, leaving the underlying source of effectiveness gains unclear. Thus, to better understand why models improve and potentially reduce the expense of training effective models, we conduct a broad ablation of sampling and distillation processes in neural ranking. We frame and theoretically derive the orthogonal nature of model geometry affected by example selection and the effect of teacher ranking entropy on ranking model optimization, establishing conditions in which data augmentation can effectively improve bias in a ranking model. Empirically, our investigation on established benchmarks and common architectures shows that sampling processes that were once highly effective in contrastive objectives may be spurious or harmful under distillation. We further investigate how data augmentation—in terms of inputs and targets—can affect effectiveness and the intrinsic behavior of models in ranking. Through this work, we aim to encourage more computationally efficient approaches that reduce focus on contrastive pairs and instead directly understand training dynamics under rankings , which better represent real-world settings. 1 Introduction Pre-trained language Models (PLMs) [ Vaswani et al., 2023 , Devlin et al., 2019 ] have been shown to be highly effective in ad-hoc ranking tasks [ Lin et al., 2021 ] . By training on large labeled datasets [ Nguyen et al., 2016 ] , they can often outperform classical term-weighting models [ Nogueira and Cho, 2019 ] . Since these early works, a key direction in improving PLM-based ranking models has been improving their data augmentation pipelines, which typically now combine “hard” negative mining [ Karpukhin et al., 2020 , Qu et al., 2021 ] (the deliberate selection of challenging non - relevant texts), and distillation of relevance estimation from an existing teacher model [ Hinton et al., 2015 , Lin et al., 2020 , Hofstätter et al., 2020 ] . While these techniques have independently demonstrated clear gains in representation learning [ Hsu et al., 2021 ] , their interaction in ranking distillation settings is poorly understood and is applied in several works with little or no ablation [ Xiao et al., 2022 , Ren et al., 2021 , Song et al., 2023 , Wang et al., 2024 ] . In Information Retrieval (IR), hard negatives are typically sampled from heuristic candidate sets scored by a preliminary model [ Karpukhin et al., 2020 , Gao et al., 2021 ] . In contrastive objectives, increasing the locality of (reducing the distance between) the candidate set creates a more challenging classification task [ Gutmann and Hyvärinen, 2010 , Ceylan and Gutmann, 2018 ] , often improving downstream effectiveness. Distillation, in turn, replaces binary labels with soft targets from a teacher [ Bucila et al., 2006 , Ba and Caruana, 2014 , Hinton et al., 2015 ] , allowing students to match or surpass larger estimators [ Pradeep et al., 2022 , Xiao et al., 2022 , Pradeep et al., 2023 ] . Contemporary systems couple the two: teachers are employed both to score and to label documents, producing multistage cascades of models that must be trained, stored, and queried at scale. Figure 1 exemplifies this ossification, showing the five - stage SentenceTransformers pipeline whose twelve cross - trained models are further filtered by a classifier [ Reimers and Gurevych, 2019 ] , sometimes upon previous iterations of each other. Such complexity is problematic. Each additional model inflates computational cost and CO 2 footprint [ Scells et al., 2022 ] and hinders reproducibility [ Wang et al., 2022 ] . From a theoretical standpoint, negative-selection heuristics influence only which instances are labelled, not the Lipschitz-geometry or teacher-entropy terms that influence generalization (cf. 2.1 ). Figure 1 : The existing SentenceTransformers [ Reimers and Gurevych, 2019 ] pipeline for hard negatives. Even naive sampling pipelines can often have three stages of prior sampling [ Song et al., 2023 , Wang et al., 2024 ] . This practise reduces our ability to attribute effectiveness gain to a particular source and more broadly replicate this process due to cross-mining of examples from different stages in the pipeline. Furthermore, existing work in increasing the “hardness” of negatives often focuses solely on the domain of a ranking [ Karpukhin et al., 2020 , Gao et al., 2021 , Qu et al., 2021 ] , either using expensive ensembling approaches as shown in Figure 1 or iteratively choosing challenging domains based on heuristics such as model uncertainty [ Xiong et al., 2021 , Zhan et al., 2021 ] . Thus, much focus is on reducing epistemic uncertainty, which may lead to a highly confident model while neglecting the irreducible aleatoric uncertainty inherent in relevance labels. Such is the appeal of distillation that there may be multiple relevant documents within a single instance. The notion of negative mining in ranking tasks is largely an artifact of contrastive objectives, often applied in representation learning. This nomenclature can be extended to functions such as ad-hoc search and reinforcement learning from human feedback [ Christiano et al., 2017 , Rafailov et al., 2023 ] ; however, the increasing use of distillation or explicit annotation means that we operate over explicit rankings instead of solely positive-negative pairs. Thus, both theoretically and empirically, we investigate common training settings controlling for heuristics governing example locality and the entropy of target distributions. We underpin empirical ablations with a generalisation bound over ranking distillation (Sec. 3) whose bias term depends exclusively on (i) the intrinsic diameter of the query manifold and (ii) the teacher’s pairwise entropy—two quantities unchanged by additional negative-mining stages. In isolating the contributions of ranking domains and target distributions in training settings, our primary contributions are two-fold: 1) We show complex pipelines to be largely identical in effectiveness to naive approaches under semi-supervision. 2) We provide clear empirical evidence of the causal factors in model effectiveness when applying data augmentation. In investigating these factors, we aim to focus research on factors of effectiveness that are appropriate given a particular search setting and reduce the spurious training and inference of multiple models. 1.1 Related Work Data Selection in Ranking Neural ranking models based on PLMs often apply contrastive objectives which benefit from the selection of similar instances, as these objectives become more challenging when examples are more similar in a geometric space but are different in terms of the target objective. Gao et al. [2021] argued that greater locality and a greater number of samples inspired by NCE [ Gutmann and Hyvärinen, 2010 ] could further enhance the benefit of localized negatives, finding that negatives sampled from more precise

Read the complete text