AttentionRetriever: Attention Layers are Secretly Long Document Retrievers

David Jiahao Fu, Lam Thanh Do, Jiayu Li, Kevin Chen-Chuan Chang · 2026

Read AttentionRetriever: Attention Layers are Secretly Long Document Retrievers by David Jiahao Fu, Lam Thanh Do, Jiayu Li, Kevin Chen-Chuan Chang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-12278.txt.

Introduction

Retrieval-augmented generation helps language models handle long documents, but most retrievers were not built for long texts. David Jiahao Fu led this work with three co-authors, presenting AttentionRetriever, a model that treats attention layers themselves as long-document retrievers. It tackles three long-text problems at once: context awareness, causal dependence, and choosing how much to retrieve.

The method joins attention signals to entity-based retrieval, building context-aware embeddings for long documents and fixing the scope of retrieval. In tests, it beats existing retrieval models on long-document sets by a large margin, while staying as efficient as dense retrieval. The result suggests that a model can find its own evidence in long texts, without a separate heavy retriever.

Catalog record

CREATORDavid Jiahao Fu, Lam Thanh Do, Jiayu Li, Kevin Chen-Chuan Chang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-12278

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AttentionRetriever: Attention Layers are Secretly Long Document Retrievers David Jiahao Fu, Lam Thanh Do, Jiayu Li, Kevin Chen-Chuan Chang Abstract Retrieval augmented generation (RAG) has been widely adopted to help Large Language Models (LLMs) to process tasks involving long documents. However, existing retrieval models are not designed for long document retrieval and fail to address several key challenges of long document retrieval, including context-awareness, causal dependence, and scope of retrieval. In this paper, we proposed AttentionRetriever, a novel long document retrieval model that leverages attention mechanism and entity-based retrieval to build context-aware embeddings for long document and determine the scope of retrieval. With extensive experiments, we found AttentionRetriever is able to outperform existing retrieval models on long document retrieval datasets by a large margin while remaining as efficient as dense retrieval models. 1 Introduction Recent advancements in Large Language Models (LLMs) 30 ; 31 ; 10 ; 16 have demonstrated strong capabilities in natural language understanding, but recent studies 22 ; 25 ; 24 have shown that LLMs still struggle to perform well on long document processing tasks due to the lost-in-the-middle problem and limited context window length, while the quadratic complexity of the attention mechanism makes processing long documents very expensive, especially for large models with hundreds of billions of parameters. In recent years, retrieval-augmented generation (RAG) 21 techniques have been widely applied to address this issue by selecting relevant information from the document with a retrieval model, which improves performance by removing distracting information and decreases processing time by shortening the input length. In the RAG pipeline, the retrieval model needs to perform long document retrieval , which requires the model to find a subset D^{\prime} from a user-provided long document D such that |D^{\prime}|\ll|D| and D^{\prime} is sufficient and necessary to answer the input query q . However, existing retrieval models are not tailored for long document retrieval and overlook the following three types of dependencies in long documents: • Contextual dependency. Since long documents are generally coherent, context is often required to resolve issues like coreference and word ambiguity, which are crucial for determining the relevance of chunks. For example, in a document discussing Chicago, the author might use "the city" to refer to "Chicago", but this reference is clear only if the context is provided. • Causal dependency. The query may involve intermediate answers from the document that are needed to reach the final answer. For the same document about Chicago, an example query would be "What was the population of Chicago when the Great Fire happened?", where the intermediate answer "the Great Fire happened in 1871" is needed to find the chunk containing the final answer. • Query dependency. Text chunks providing background information, like the one containing "the Great Fire happened in 1871" in the previous example, are also important to answering the query and should be retrieved. However, these chunks might receive low similarity scores because they are not very relevant to the query, which asks about "the population of Chicago". Therefore, it is necessary to accurately decide the scope of retrieval in long document retrieval tasks. Modeling the first two dependencies requires a more advanced retrieval model that can build context-aware representations and update the embeddings as additional contextual information is available. We found that the attention layers in transformer models perfectly match both requirements. Since attention layers calculate the representations of each token by aggregating information from other tokens, they are essentially cross-encoders that embed contextual information into the representation of each token, providing more abundant semantic information compared to existing embedding-based retrieval models. Furthermore, as the representations are propagated through layers, they are also dynamically adjusted based on contextual information gathered in previous layers to encode causal dependencies. It is also intuitive to use attention layers as retrievers because the attention operations are essentially calculating similarity scores. In each attention layer of the transformer model, the attention score assigned to the j -th token by the i -th token is calculated as the weighted dot product between the key vector k_{j} and the query vector q_{i} , which is identical to similarity calculation of embedding models. Moreover, attention computation is performed on two sets of embeddings q and k , which allows the attention layers to perform a broader range of tasks other than semantic similarity search by adjusting the embeddings. However, training a transformer model for retrieval is very expensive. 44 found that the last layer in the Qwen-2 model shows high retrieval accuracy without any additional training, implying the possibility of directly employing pretrained LLMs to estimate relevance with attention scores. However, their experiments were very limited and the findings might not generalize to other attention layers and other LLMs. More importantly, since pretrained LLMs suffer from accuracy and efficiency issues with long context, it is also crucial to find out whether attention scores can be efficiently calculated and still achieve high retrieval accuracy with long context. Therefore, we conducted careful analysis (Section 3 ) to verify if attention layers in pretrained LLMs can be effective and efficient training-free retrievers. Our analysis showed that attention scores are more precise than outputs in collecting relevant information and suffer less from the lost-in-the-middle problem , validating the effectiveness of using LLMs for retrieval tasks despite that they face various issues in long context tasks. Moreover, existing attention approximation methods for LLMs can be directly applied to this approach, making it possible to process long documents with arbitrary lengths. Furthermore, we found that employing pretrained LLMs with around 3 billion parameters as retrievers is already able to achieve impressive performance, which eliminates the need for using larger LLMs and improves efficiency. However, attention scoring alone is still insufficient to model the third dependency. Background information is still unlikely to receive high attention scores in layers with high retrieval accuracies because it is already embedded into the representations before these layers. To obtain a better estimate of the scope of retrieval, it is essential to additionally consider text chunks that are not immediately relevant but provide background information. Since each piece of background information typically focuses on one entity, and it should be included as part of the retrieval result only if the entity is relevant to the input query, we believe an entity graph structure could help to determine the retrieval scope precisely by connecting text chunks through entities and finding the entities relevant to the query during retrieval to discover hidden background information. In contrast to knowledge graphs, entity graphs are much easier and more efficient to construct, without the need to extract relationships between entities. Figure 1: Overview of AttentionRetriever. Based on these findings and analysis, we proposed AttentionRetriever, a novel retrieval model that builds context-aware embeddings with pretrained LLMs and decides the scope of retrieval through entity-based retrieval, as summarized in Figure 1 . During retrieval, we leverage LLMs to process the long document and the query together, adopting the attention maps at layers that show high retrieval accuracies to estimate the relevance score of each text segment, which is combined with embedding-based similarity scoring for more precise retrieval. To decide the scope of retrieval, we find the desired entities by ranking them by the scores of

Read the complete text