ColBERT-Att : Late-Interaction Meets Attention for Enhanced Retrieval

Raj Nath Patel, Sourav Dutta · 2026

Read ColBERT-Att : Late-Interaction Meets Attention for Enhanced Retrieval by Raj Nath Patel, Sourav Dutta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-25248.txt.

Introduction

Patel and Dutta add attention weights to a late-interaction search method. ColBERT-style search scores each query word against document words late in the pipe, which is fast and sharp. But it ignores how much each word mattered inside the model. ColBERT-Att folds those attention weights into the match scores. The change aims to weigh key word pairs more than filler pairs. Recall improves on MS-MARCO. Gains also show across wide BEIR and LoTTE test sets.

Catalog record

CREATORRaj Nath Patel, Sourav Dutta
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16437
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.25248

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ColBERT-Att : Late-Interaction Meets Attention for Enhanced Retrieval Raj Nath Patel, Sourav Dutta Abstract Vector embeddings from pre-trained language models form a core component in Neural Information Retrieval systems across a multitude of knowledge extraction tasks. The paradigm of late interaction , introduced in ColBERT, demonstrates high accuracy along with runtime efficiency. However, the current formulation fails to take into account the attention weights of query and document terms, which intuitively capture the “importance” of similarities between them, that might lead to a better understanding of relevance between the queries and documents. This work proposes ColBERT-Att , to explicitly integrate attention mechanism into the late interaction framework for enhanced retrieval performance. Empirical evaluation of ColBERT-Att depicts improvements in recall accuracy on MS-MARCO as well as on a wide range of BEIR and LoTTE benchmark datasets. 1 Introduction & Background Semantic similarity or relevance between queries and documents using high-dimensional dense vector representations of texts, from large language models, has become ubiquitous in Information Retrieval (IR) and also forms a core component in Retrieval Augmented Generation (RAG) 6 . Retrieval of relevant documents or information has transitioned from lexical and text matching (e.g., BM25 10 ) to semantic retrieval based on neural models (e.g., ColBERT 5 ) capturing the semantics and context of information contents. Traditional systems like BM25 using simplistic text matching (e.g., TF-IDF 14 ; 11 measure) based on sparse encoding and corpora statistics, failed to capture similarities beyond surface form equivalence. SPLADE 2 provides an explicit sparsity regularization and a log-saturation effect for obtaining term weights, eliminating the need of statistics and hyper-parameters. However, it suffers from language dependency along with expensive memory and compute requirements. BM42 17 aims to combine the strengths of lexical matching and attention mechanism from language models. The use of document token attention weights as a proxy to documents term importance was shown to be effective. Orthogonally, neural IR methods 4 ; 9 ; 18 ; 3 ; 7 allow semantic and contextual matching by encoding queries and documents into single-vector dense representations from language models to gauge the relevance between queries and documents. Late interaction strategy used in ColBERT, produces multi-vector representations at token or subtoken level granularity, and computes fine-grained relevance scores between query and document tokens using efficient and scalable token-level computations. Specifically, the original ColBERT architecture computes the maximum cosine similarity (i.e., MaxSim operation) between a query token and all the document tokens, using the token-level dense representations. Finally, a summation of the maximum similarities over all the query tokens is computed to obtain the final relevance scores between query and documents. Consider, \mathcal{E}_{q_{i}} and \mathcal{E}_{d_{j}} to denote the embeddings of the i^{th} token of query \mathcal{Q} and the j^{th} token in document \mathcal{D} respectively. Mathematically, the score is thus computed as, \displaystyle\mathcal{S}_{\mathcal{Q},~\mathcal{D}}=\sum_{q_{i}\in\mathcal{Q}}\max_{d_{j}\in\mathcal{D}}\mathcal{E}_{q_{i}}\cdot\mathcal{E}_{d_{j}} (1) ColBERT has been shown to perform extremely well on retrieval tasks by leveraging the semantic expressiveness of language model embeddings, along with the ability to pre-compute document representations for speeding up query processing latency. To improve upon the accuracy and memory footprint of the late interaction architecture, vector compression techniques and denoising training strategy were proposed in ColBERTv2 13 . The use of centroid interaction and pruning approach were introduced in ColBERTv2 {}_{\text{{PLAID}}} 12 , to improve the search latency with comparable performance. Rotary positional embeddings 15 and advanced activation functions for enhanced retrieval performance were recently incorporated in ModernColBERT 1 . Motivation. Observe that the MaxSim operation in ColBERT does not explicitly factor in the importance of either the query or document terms. As such, all term matches between query and documents are considered to be of equal importance, which is not necessarily true and might degrade the overall performance. Although, vector embeddings obtained from language models implicitly leverage attention weights, we posit that explicit incorporation of terms importance via attention mechanism within the relevance score computation could be beneficial. As an intuition of how attention weights can enable better understanding of query-document relevance , consider the following toy example. Assume a query and candidate documents as: \mathcal{~Q} : Who is going to study ? \mathcal{D}_{1} : Alice is walking to school . \mathcal{D}_{2} : Bob is going to buy apples . \mathcal{D}_{3} : Only studying makes Jack a dull boy . Here, we highlight probable key terms (underlined) within the query and documents, which would ideally have higher attention weights from a contextual language model. Observe, the phrase “is going to” in \mathcal{Q} has a high (embedding based cosine) similarity with \mathcal{D}_{2} , since it is present in both the texts. However the attention weights of these terms are relatively low in \mathcal{Q} , as they are not important to the overall context and intent of the query. Thus, incorporation of query term attention would effectively and accurately reduce the relevance between \mathcal{Q} and \mathcal{D}_{2} . Similarly, the term “studying” having low attention weight in document \mathcal{D}_{3} would diminish its relevance to the query, in spite of a high similarity match with the term “study” in \mathcal{Q} . On the other hand, although the terms “study” and “school” in \mathcal{Q} and \mathcal{D}_{1} resp. are related (having moderate cosine similarity), the high query and document term attention weights would boost the relevance score between the query and document – thus accurately retrieving \mathcal{D}_{1} for query \mathcal{Q} . Thus, incorporation of query and document attention weights within the late-interaction framework could potentially improve the overall retrieval performance, as proposed here in ColBERT-Att . 2 ColBERT-Att Training Consider a query \mathcal{Q} and a document \mathcal{D} to be composed of n and m tokens respectively. Thus, \mathcal{Q}=\{q_{1},q_{2},\cdots,q_{n}\} and \mathcal{D}=\{d_{1},d_{2},\cdots,d_{m}\} . The relevance score of the document to the query ( \mathcal{S}_{\mathcal{Q},~\mathcal{D}} ) is then computed as, \displaystyle\mathcal{S}_{\mathcal{Q},~\mathcal{D}}=\sum_{i=1}^{n}e^{~\mathcal{A}_{q_{i}}}\cdot\max_{j=1}^{m}(\mathcal{E}_{q_{i}}\odot\mathcal{E}_{d_{j}})\cdot(e^{~\mathcal{A}_{d_{w}}})^{{}^{\delta}} (2) where, \mathcal{E} and \mathcal{A} represents the corresponding vector embeddings and attention weights respectively. Further, \odot denotes the cosine similarity operator, and the document token that depicts the highest similarity to the query token q_{i} is represented as d_{w}=\argmax_{j\in[1,\cdots,m]}~(\mathcal{E}_{q_{i}}\odot\mathcal{E}_{d_{j}}) . Finally, \delta signifies a document length based attention weight regularizer , which we discuss later in this section. In other words, we augment the MaxSim operation of Eq. ( 1 ) with the corresponding query token attention weight along with the associated document token attention weight that depicts the highest (cosine) similarity to the query token. Since the attention weights are typically small values, we accentuate their values (and relative differences) by taking the exponent. Following the original framework of ColBERT, we consider the queries to comprise 32 tokens and documents to be represented by 300 tokens (including special and mask tokens) for most datasets unless specified otherwise. To obtain our ColBERT-Att model, we trained ColBERTv2 {}_{\text{{PLAID}}} (obtained from https://github.com/stanford-futuredata/ColBERT ) with the modified objective of Eq. ( 2 ) (with \delta=1 ) using the official train queries and corresponding positive/negative triples of MS-MARCO dataset. Training was performed for 1M steps with default parameter settings, and the best checkpoint was considered. The document token embeddings and attention weights are computed offline and stored as a pre-processing step, while the query token embeddings and attentions are obtained on-the-fly during inference. Observe, that the attention weights are technically free (in terms of compute), as they are an inherent artifact of the encoding

Read the complete text