Working Notes on Late Interaction Dynamics: Analyzing Targeted Behaviors of Late Interaction Models

Antoine Edy, Max Conti, Quentin Macé · 2026

Read Working Notes on Late Interaction Dynamics: Analyzing Targeted Behaviors of Late Interaction Models by Antoine Edy, Max Conti, Quentin Macé (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26259.txt.

Introduction

Edy, Conti, and Macé study two quirks of late interaction search models. First, they test a length bias that can arise when multi-vector scores are pooled. Second, they look at the spread of similarity scores past the best match used by the MaxSim rule. Tests on NanoBEIR show the bias holds for causal models, and can hit two-way models in extreme cases. They also find no strong trend past the top token, which supports the MaxSim rule.

Catalog record

CREATORAntoine Edy, Max Conti, Quentin Macé
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE14837
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26259

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Working Notes on Late Interaction Dynamics: Analyzing Targeted Behaviors of Late Interaction Models Antoine Edy, Max Conti, Quentin Macé Abstract While Late Interaction models exhibit strong retrieval performance, many of their underlying dynamics remain understudied, potentially hiding performance bottlenecks. In this work, we focus on two topics in Late Interaction retrieval: a length bias that arises when using multi-vector scoring, and the similarity distribution beyond the best scores pooled by the MaxSim operator. We analyze these behaviors for state-of-the-art models on the NanoBEIR benchmark. Results show that while the theoretical length bias of causal Late Interaction models holds in practice, bi-directional models can also suffer from it in extreme cases. We also note that no significant similarity trend lies beyond the top-1 document token, validating that the MaxSim operator efficiently exploits the token-level similarity scores. [[email protected], ] *1 1 [[email protected], ] 1 [[email protected], ] keywords Information Retrieval ,Late-interaction ,Multi-Vector Retrieval ,Causal Encoders ,Bi-directional Encoders † † copyrightyear: 2026 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: Late Interaction Workshop (LIR) @ ECIR 2026, April 02, 2026. Colocated with ECIR 2026. † † address: Illuin Technology † † corresp: Corresponding author. † † These authors contributed equally. 1 Introduction Neural late-interaction retrieval models, such as ColBERT 5 , use a token-level interaction while computing similarity between text passages. While this approach allows for finer semantic matching between queries and documents, some of its underlying dynamics are yet to be thoroughly studied. In these notes, we analyze two key behaviors that provide elements for a better understanding of Late Interaction performance: (a) Length bias: Causal encoders, when used with multi-vector MaxSim scoring, exhibit a monotonic bias that favors longer chunks, regardless of their true relevance. (b) Similarity distribution: Given a query token, the MaxSim operator is insensitive to similarity scores of document tokens beyond the highest, collapsing information to a single maximum value. We perform small-scale experiments on the NanoBEIR 6 benchmark to bring more insights into how current state-of-the-art models behave along these axes. 2 Length Bias In Multi-Vector Retrieval In this section, we explore the length bias that arises in late-interaction retrieval frameworks. We highlight two key observations: multi-vector causal models appear to suffer from a strict, monotonic length bias, and while bi-directional architectures theoretically avoid this flaw, empirical insights suggest they remain sensitive to length disparities at extreme margins. Details on the experimental setup are available in Appendix A . 2.1 Theoretical Motivation Let a chunk c be a sequence of tokens represented by contextualized embeddings. In late-interaction retrieval, the MaxSim score 5 between a query q and a chunk c is defined as: S_{q,c}=\sum_{i\in[|E_{q}|]}\max_{j\in[|E_{c}|]}E_{q_{i}}\cdot E_{c_{j}}^{T} where E_{q} and E_{c} are the respective sets of query and chunk embeddings. When utilizing a causal encoder with a multi-vector representation, appending tokens to a chunk yields a strict superset of embeddings. As a result, the maximum inner product for each query token can only increase or remain constant. This dynamic introduces a theoretical monotonic length bias that artificially favors longer chunks regardless of their true relevance. Bi-directional and single-vector models theoretically avoid this strict bias. In bi-directional models, appending new tokens alters the attention context for all preceding tokens, allowing scores to naturally decrease if the semantic focus is diluted. Similarly, single-vector models aggregate tokens into a fixed-length representation that does not inherently benefit from added tokens. However, to understand their practical robustness to length differences, we complement this theoretical intuition with an empirical analysis. 2.2 Multi-Vector Architectures Induce A Length Bias Figure 1 : Mean length comparison between the retrieved false positive chunks, the relevant ground-truth documents, and the global corpus average. Queries are grouped into quantiles on the x-axis based entirely on the average token length of their corresponding relevant chunks. Figure 1 isolates the impact of the pooling mechanism in causal architectures by comparing a multi-vector model ( jina-embeddings-v4 4 ) with a single-vector dense model ( Qwen3-Embedding-4B 9 ). Queries on the horizontal axis are partitioned into quantiles based on the token length of their true relevant documents. For each quantile, we compare the mean length of retrieved false positives against the true relevant documents, as well as the global corpus mean (199 tokens). While false positives are statistically expected to be slightly longer on average due to the inherently wider semantic scope of longer texts, this length gap should ideally remain marginal. However, the multi-vector causal model disproportionately retrieves false positives that are significantly longer than the relevant documents. Conversely, the single-vector causal model tracks the relevant document length much more closely. This confirms that within causal architectures, the multi-vector setup is the primary driver of length bias. 2.3 Bi-Directional Models Mitigate But Do Not Eliminate Bias (a) jina-embeddings-v4 (b) Qwen3-Embedding-4B (c) GTE-ModernColBERT-v1 (d) ColBERT-Zero Figure 2 : Expected decrease in retrieval performance (nDCG@10) when a chunk of a specific length is added to the corpus. Chunks are categorized into equal-sized quantile bins by token length on the x-axis. The solid line plots the average nDCG penalty incurred by the presence of a chunk from that bin, evaluated against a random baseline (dashed line) and its 90% confidence interval (shaded area). Having identified the multi-vector setup as a primary driver of length bias in causal architectures, we now investigate whether bi-directional attention mechanisms can effectively neutralize this issue. We evaluate the expected unconditional decrease in retrieval performance (nDCG) when a chunk of a specific length is added to the corpus. To isolate the effect of length, we compute a random baseline and a 90% confidence interval using a permutation test that assumes no correlation between chunk length and retrieval harm. Any deviation outside this interval demonstrates a statistically significant length bias, indicating that adding chunks of that size disproportionately harms ranking quality compared to random chance. The results reveal distinct architectural behaviors. The causal multi-vector model (Figure 2(a) ) exhibits a near-monotonic length bias; adding longer chunks consistently translates to a greater expected decrease in nDCG. In contrast, the single-vector dense model (Figure 2(b) ) displays no significant length bias, remaining safely within the random baseline and corroborating the findings of the previous section. Notably, while bi-directional multi-vector models successfully dampen the aggressive bias of causal models, they remain vulnerable at length extremes (Figures 2(c) and 2(d) ). For these models, adding unusually short chunks is significantly less harmful than expected by random chance, whereas introducing exceptionally long chunks disproportionately degrades overall ranking quality. Thus, while bi-directional attention refines token representations, it remains unable to fully recover the systemic length bias introduced by the MaxSim operation across substantial token variations. 3 Similarity Distribution: What Happens Beyond The Top-1 Document Token The MaxSim operator described in subsection 2.1 , by construction, only considers the single most similar document token for each query token, ignoring the number and density of relevant tokens in a chunk. This could lead to hypersensitivity to the highest similarity score of document tokens. Intuitively, a single query token that has several strong matches in a document A is more similar to it than to a document B, where only one token has a high similarity, a

Read the complete text