Pathway to Relevance: How Cross-Encoders Implement a Semantic Variant of BM25
Meng Lu, Catherine Chen, Carsten Eickhoff · 2025
Read Pathway to Relevance: How Cross-Encoders Implement a Semantic Variant of BM25 by Meng Lu, Catherine Chen, Carsten Eickhoff (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04645.txt.
Introduction
Meng Lu, Catherine Chen, and Carsten Eickhoff present a 2025 paper on how a search ranking model decides that a document matches a query. Cross-encoders read query and document together and score relevance, but their inner logic was a black box. The team opened one up and traced its signals layer by layer.
Early and middle layers pull out classic clues: how often query terms appear, and how rare they are across the collection. Later layers blend those clues much like BM25, the famous ranking formula from classical search. So the neural model quietly rebuilds a semantic version of the old recipe. Knowing which parts carry which signal opens doors to clearer, safer, and more scalable ranking systems.
Catalog record
| CREATOR | Meng Lu, Catherine Chen, Carsten Eickhoff |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-04645
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pathway to Relevance: How Cross-Encoders Implement a Semantic Variant of BM25 Meng Lu, Catherine Chen, Carsten Eickhoff Abstract Mechanistic interpretation has greatly contributed to a more detailed understanding of generative language models, enabling significant progress in identifying structures that implement key behaviors through interactions between internal components. In contrast, interpretability in information retrieval (IR) remains relatively coarse-grained, and much is still unknown as to how IR models determine whether a document is relevant to a query. In this work, we address this gap by mechanistically analyzing how one commonly used model, a cross-encoder, estimates relevance. We find that the model extracts traditional relevance signals, such as term frequency and inverse document frequency, in early-to-middle layers. These concepts are then combined in later layers, similar to the well-known probabilistic ranking function, BM25. Overall, our analysis offers a more nuanced understanding of how IR models compute relevance. Isolating these components lays the groundwork for future interventions that could enhance transparency, mitigate safety risks, and improve scalability. 1 Introduction Information retrieval (IR) is the subfield of NLP that aims to rank a collection of documents by their relevance to a specific query. Traditional ranking strategies have long relied on probabilistic models grounded in intuitive heuristics, like BM25 ( Robertson and Zaragoza, 2009 ) , to estimate relevance. BM25 leverages term frequency (TF) and inverse document frequency (IDF) to rank documents effectively, achieving strong performance across various tasks. Its simplicity and inherent interpretability have made it a cornerstone of traditional IR systems. Inspired by BM25’s success, earlier neural IR models ( Pang et al., 2016 ; Guo et al., 2016 ) were purposefully designed to emulate BM25’s principles. These models incorporate explicit components for semantic TF and IDF computations, blending neural architectures with established IR heuristics to improve relevance estimation. However, the advent of transformer-based models revolutionized the field of IR. These models, trained end-to-end on large numbers of query-document pairs ( Nguyen et al., 2016 ; Thakur et al., 2021 ) , excel at extracting context-dependent semantic signals for ranking tasks. By leveraging multi-headed attention and vast parameter spaces, transformers Vaswani et al. (2017) capture nuanced relationships between query and document terms that go beyond traditional heuristic-based approaches. Despite their superior performance, these models come with significant trade-offs: their complexity and lack of interpretability make it challenging to understand their internal mechanisms. This raises fundamental questions: how do these models assess relevance? Do they rely on established IR principles such as TF and IDF, or do they draw on entirely different expressions of relevance? In this work, we build upon previous correlational studies ( Choi et al., 2022 ; Zhan et al., 2020 ; Formal et al., 2021 ; Formal et al., 2022 ; MacAvaney et al., 2022 ) by employing mechanistic interpretability methods ( Nanda, 2022 ; Elhage et al., 2021 ; Olsson et al., 2022 ; Meng et al., 2024 ; Wang et al., 2022 ; Pearl, 2022 ) to address these questions. Concretely, we analyze a BERT-based IR model (i.e., a cross-encoder) to isolate multiple relevance signals beyond exact match TF. 1 1 1 All code and resources are available at: https://github.com/mlu108/CrossEncoderBM25 Our main findings are: (1) We identify attention heads that extract and process BM25-like components, including a semantic version of term frequency (soft-TF), term saturation, and document length (§ 4.1 -§ 4.3 ). (2) We find evidence that IDF information largely already exists in the largest singular value of the embedding matrix and can be manipulated to later control term importance (§ 4.4 ). (3) We confirm that heads in later layers aggregate all these relevance signals in a BM25-style manner by defining a linear approximation of the hypothesized relevance computation and evaluating its ability to reconstruct cross-encoder’s predicted scores, confirming that our circuit captures the core mechanism of relevance computation (§ 5 ). Overall, our mechanistic analysis uncovers insights on how IR models determine relevance, paving the way for targeted model editing to boost performance, enable personalization, and mitigate bias in transformer - based IR. 2 Background and Related Work 2.1 Axioms and BM25 BM25 Component Axiom Perturbation Axiom-Based Diagnostic Dataset Examples ( b : baseline doc, p : perturbed doc) TF ( soft-TF ) TFC1 Fang et al. (2004) : Prefer documents with more query term occurrences. Given a baseline document, we perturb it by appending one more selected query term. b : Quebec is a small city in Canada. p : Quebec is a small city in Canada. Quebec . STMC1 Fang and Zhai (2006) : Prefer documents with semantically similar terms. Given a baseline document, we append a semantically similar term - the synonym with highest embedding cosine similarity with query term from 20 candidates generated by GPT-4o. b : Quebec is a small city in Canada. p : Quebec is a small city in Canada. Toronto . IDF TDC Fang et al. (2004) : Prefer documents with more discriminative query terms. N/A N/A k TFC2 Fang et al. (2004) : Additional occurrences yield smaller improvements. We create a baseline document using GPT-4o to create five relevant sentences starting with selected query term’s pronoun and perturb by incrementally restoring the term. b : It is in Canada. It is fun. p 1 : Quebec is in Canada. It is fun. p 2 : Quebec is in Canada. Quebec is fun. b LNC1 Fang et al. (2004) : Penalize longer documents for non-relevant terms. Given a baseline document, we create five perturbations by sequentially appending an increasing number of random sentences from a non-relevant query in the base dataset. b : Quebec is in Canada. p 1 : Quebec is in Canada. Road not taken is written by Frost. p 2 : Quebec is in Canada. Road not taken is written by Frost. Happiness is a Butterfly. Table 1: Mapping of BM25 components to IR axioms. Note: Since IDF is a distinct component in BM25 (unlike term saturation and document length, which are tied to TF), we use alternative methods to isolate it (§ 4.4 ). Axiomatic IR constructs formal desiderata, or axioms, outlining specific properties that an effective ranking model should satisfy ( Bruza and Huibers, 1994 ) . For example, the TFC1 axiom ( Fang et al., 2004 ) states that documents that contain more query terms should be ranked higher, and the TDC axiom ( Fang et al., 2004 ) states that documents containing query terms with higher inverse document frequency (IDF) should receive higher scores. These axioms provide a theoretical foundation for understanding and developing ranking functions by formalizing human-interpretable and intuitive notions of relevance into mathematical constraints. BM25 is a widely used probabilistic ranking function that exemplifies these axiomatic principles by ranking documents based on term frequency (TF), inverse document frequency (IDF), term saturation, and document length. It is defined as: \sum_{t\in q}\text{IDF}(t)\cdot\frac{\text{TF}(t,d)\cdot(k_{1}+1)}{\text{TF}(t,d)+k_{1}\cdot(1-b+b\cdot\frac{|d|}{\text{avgdl}})} (1) where q is the query, d is the document, t is a query term, k_{1} and b are hyperparameters controlling term saturation and length normalization, respectively. |d| is the length of document d , and avgdl is the average document length in the corpus. Intuitively, BM25 computes a relevance score by multiplying each query term’s term frequency (TF) in the document with its inverse document frequency