Understanding Matching Mechanisms in Cross-Encoders DOI: XXXXXXX.XXXXXXX Conference: Workshop on Explainability in Information Retrieval at SIGIR 2025; July 17, 2025; Padova, Italy

Mathias Vast, Basile Van Cooten, Laure Soulier, Benjamin Piwowarski · 2025

Read Understanding Matching Mechanisms in Cross-Encoders DOI: XXXXXXX.XXXXXXX Conference: Workshop on Explainability in Information Retrieval at SIGIR 2025; July 17, 2025; Padova, Italy by Mathias Vast, Basile Van Cooten, Laure Soulier, Benjamin Piwowarski (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-14604.txt.

Introduction

Mathias Vast, Basile Van Cooten, Laure Soulier, and Benjamin Piwowarski peek inside the ranking models called cross-encoders, in this 2025 workshop paper. The catalog title carries venue metadata, so the title follows the head of the text. These models read a query and a document together and score their match, topping search leaderboards while hiding how they match.

Most past probing asked only what sways the score. This team digs into the matching steps themselves with plain tools. Attention analysis shows a few heads carry the causal weight of the process. The team then offers a reading of how match detection works inside the network. They argue simple methods already yield real insight, no heavy machinery needed.

Catalog record

CREATORMathias Vast, Basile Van Cooten, Laure Soulier, Benjamin Piwowarski
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE10 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-14604

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Understanding Matching Mechanisms in Cross-Encoders DOI: XXXXXXX.XXXXXXX Conference: Workshop on Explainability in Information Retrieval at SIGIR 2025; July 17, 2025; Padova, Italy Mathias Vast, Basile Van Cooten, Laure Soulier, Benjamin Piwowarski Abstract . Neural IR architectures, particularly cross-encoders, are highly effective models whose internal mechanisms are mostly unknown. Most works trying to explain their behavior focused on high-level processes (e.g., what in the input influences the prediction, does the model adhere to known IR axioms) but fall short of describing the matching process. Instead of Mechanistic Interpretability approaches which specifically aim at explaining the hidden mechanisms of neural models, we demonstrate that more straightforward methods can already provide valuable insights. In this paper, we first focus on the attention process and extract causal insights highlighting the crucial roles of some attention heads in this process. Second, we provide an interpretation of the mechanism underlying matching detection. 1 1 1 Code is available here: https://git.isir.upmc.fr/mat_vast/sigir25-matching-signals . \acmCodeLink https://git.isir.upmc.fr/mat_vast/sigir25-matching-signals \acmArticleType Review 1. Problem statement The recent surge in interpretability research stems from a dual motivation: to understand how neural networks accomplish complex tasks and to identify their inherent limitations. In IR, a key research direction is explainability – i.e., understanding which input components and model elements contribute to predictions. Previous studies have made significant strides in unraveling IR model architectures. For instance, Zhan et al. 2020 Zhan et al. 2020 studied cross-encoders and elucidated the role of "[CLS]" and "[SEP]" in the "no-op" operation and demonstrated that relevance prediction is a multi-stage process, involving query and document contextualization, the detection of interaction signals, and finally their composition. Contemporary to this work, Jiang et al. 2021 Jiang et al. 2021 applies attribution-based method Schulz et al. 2020 to identify the tokens that contribute the most to the final prediction. Notably, they find that MonoBERT Nogueira & Cho 2019 still relies on lexical matching but is also able to perform semantic matching, and the "[CLS]" only starts to aggregate relevance signals after layer 16. Based on these insights, Chen et al. 2024 Chen et al. 2024 further revealed the nuanced roles of specific attention heads in the detection of token duplications and the composition of relevance signals within dual embedding models. These findings suggest a sophisticated, layered information processing mechanism. Our research extends these investigations by providing a more in-depth analysis of how neural ranking models construct relevance signals. Specifically, we examine the interactions between queries and documents by tracing the information flow through the network. We 1) extract causal insights from the attention process, highlighting the critical roles of specific attention heads in matching, and 2) characterize the nature of these matches. Subsequently, we conduct a deep dive into the internal mechanisms of these matching heads to understand their behavior. (a) Impact on nDCG@10 of the ablation of a direction up to each layer. (b) Average diff. in the attention between relevant and easy negatives. (c) Comparison of attention distributions between relevant and easy negative passages. Figure 1 . Summary of the experiments on the attention of Query and Document (layer \ell /head h ). 2. Methods To do so, we study cross-encoders with an ablation study and the analysis of attention matrices. To gain a better understanding of the transfers of information, we focus on their directions across input parts. We split each query-passage pair into 5 distinct parts: the CLS token, the query Q , the first separator token SEP1 , the document D , and the last separator token SEP2 and look at the attention of the input’s part X to the input’s part Y , a direction , within the attention weight matrix or equivalently, the information transmitted by the input’s part Y to the input’s part X with X,Y\in\{CLS,Q,SEP1,D,SEP2\} . We use the Deep Learning tracks from TREC between 2019 and 2022 Craswell et al. 2021 ; Craswell et al. 2022 ; Craswell et al. 2023 ; Craswell et al. 2020 to conduct our experiments. The annotations have 4 degrees of relevance r\in\{0..3\} . Because documents labeled as "irrelevant" ( r=0 ) are still semantically related (a search engine retrieved them), we also study the behavior of MonoBERT in the case where the document is irrelevant and not semantically related, by randomly sampling a document in the dataset. We call these documents easy negatives , or r=-1 , as opposed to the hard negatives , or r=0 . This nuance between negatives helps us identify the different types of matching, as we hypothesize that semantic matches are more likely to occur in hard negatives than easy negatives. 3. Main Results Table 1 . nDCG@10 results of the ablations. Results in bold indicate a decrease compared to the Original model, and * indicates a decrease with statistical significance ( p\leq 0.05 ) under the two-tailed Student’s t-test. \nleftarrow CLS Query SEP1 Doc. SEP2 All CLS 0.82 0.79 0.81 0.81 0.81 0.48* Query 0.82 0.78 0.81 0.80 0.81 0.66* SEP1 0.81 0.81 0.81 0.81 0.81 0.81 Doc. 0.81 0.79 0.81 0.67* 0.82 0.62* SEP2 0.81 0.81 0.81 0.81 0.81 0.81 Original perf. = 0.81 / Random ranking = 0.48 (nDCG@10). With our ablation study, we first validate the importance of the document’s contextualization and query-to-document information transfers (Fig. 1(a) ). Our results, however, question Zhan et al. 2020 Zhan et al. 2020 conclusion on the importance of document-to-query transfers (Table 1 ). Second, we study the attention weights between the different parts of the input. We confirm the role in the "no-op" operation of the "[SEP]" and further show that the "[CLS]" might play a similar role through the early to the middle layers. It further unveils attention heads specialized in detecting interactions between queries and relevant passages. To deepen this observation, we compare the attention weight distributions between documents and queries (Fig. 1(b) and 1(c) ) hints at the following behavior: 1) From early to middle layers, these interactions are matching between terms of the query and document ( syntactic matching ); 2) From middle layers, matching occurs between semantically-contextualized tokens ( semantic matching ). Finally, we explain how this matching behavior takes place by analyzing some attention heads A_{h}=K_{h}^{\top}Q_{h} . The logit of the probability that token i attends to j is x_{j}^{\top}A_{h}x_{i} . If we suppose that the representation of the i^{th} token of the query and the j^{th} token of the document (at any layer \ell in the model) can be expressed as E_{i}=q+x_{i}+a_{i} and E_{j}=d+y_{j}+b_{j} , where q and d respectively represents the part of the embedding of queries and documents shared by every query and document, x_{i} and y_{j} their semantic/lexical information, and a_{i} and b_{j} some residual information. Then, for a given head h , the attention logit for E_{i} on E_{j} can be written as: (1) \displaystyle(E_{i})^{\top}A_{h}E_{j} \displaystyle=(q+x_{i}+a_{i})^{\top}A_{h}(d+y_{j}+b_{j}) (2) \displaystyle=q^{\top}A_{h}d+x_{i}A_{h}y_{j}+\epsilon with \epsilon\approx 0 . To validate this hypothesis, we plot the difference between q^{\top}A_{h}d-d^{\top}A_{h}d , which should be greater than 0 when query-document matching is sought, against the propensity of matching tokens with the same meaning, i.e. where x_{i} and x_{j} are close. For the latter, we measure to which extent the left and right singular vectors of the Singular Value Decomposition of A_{h} are similar, i.e. with A_{h}=\sum\sigma_{k}u_{k}v_{k}^{\top} , we compute

Read the complete text