AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation
Lvzhou Luo, Yixuan Cao, Ping Luo · 2025
Read AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation by Lvzhou Luo, Yixuan Cao, Ping Luo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17486.txt.
Introduction
Lvzhou Luo, Yixuan Cao, and Ping Luo trim the context fed to language models in a 2025 arXiv paper. Retrieval-augmented generation grounds answers in outside documents, but the retrieved stack often holds irrelevant text that drags quality down. Compression can strip the chaff, yet fixed rates fit some queries badly and multi-document facts can be lost.
Their framework, AttnComp, lets the model's own attention scores judge what matters. A Top-P rule keeps the smallest set of documents whose total attention clears a set threshold, so the rate adapts to each case. The method stays fast and merges facts across documents. It also estimates answer confidence from the overall relevance of what was retrieved, so users can gauge how far to trust a reply. In tests, AttnComp beat rival compressors and even uncompressed baselines on accuracy.
Catalog record
| CREATOR | Lvzhou Luo, Yixuan Cao, Ping Luo |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-17486
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AttnComp: Attention-Guided Adaptive Context Compression for Retrieval-Augmented Generation Lvzhou Luo, Yixuan Cao, Ping Luo Abstract Retrieval-augmented generation improves the factual accuracy of Large Language Models (LLMs) by incorporating external context, but often suffers from irrelevant retrieved content that hinders effectiveness. Context compression addresses this issue by filtering out irrelevant information from context before LLM generation. However, existing methods struggle to adaptively adjust compression rates for different context, maintain low latency and integrate information across multiple documents. To overcome these limitations, We introduce AttnComp, an adaptive, efficient and context-aware compression framework. By leveraging the attention mechanism of LLMs to identify relevant information, AttnComp employs a Top-P compression algorithm to retain the minimal set of documents whose cumulative attention weights exceeds a predefined threshold. In addition to compression, AttnComp estimates response confidence by assessing the overall relevance of the retrieved content, enabling users to gauge response reliability. Experiments demonstrate that AttnComp outperforms existing compression methods and uncompressed baselines, achieving higher accuracy with substantial compression rates and lower latency. † † footnotetext: * Corresponding author: Yixuan Cao and Ping Luo. 1 Introduction Figure 1: Illustration of the AttnComp Framework. AttnComp consists of two stages: Stage 1 involves attention computation, where attention weights are calculated from the query (q) to the context composed of the instruction (Ins) and documents (d); Stage 2 applies a Top-P compression algorithm to select the most relevant documents and generate a confidence score for the RAG response. Three cases are illustrated: (1) Documents 1 and 3 are relevant and retained; (2) Only Document 2 is relevant and retained; (3) All documents are irrelevant and filtered out. Retrieval-Augmented Generation (RAG) enhances the factual accuracy and reliability of Large Language Models (LLMs) in knowledge-intensive tasks by integrating retrieved context into their generation process ( Lewis et al. 2020 ; Borgeaud et al. 2022 ; Izacard et al. 2023 ; Ram et al. 2023 ; Xu et al. 2023b ) . However, practical RAG applications often grapple with retrieved content containing substantial irrelevant information, even entirely unrelated to the query ( Sauchuk et al. 2022 ) . This gives rise to three primary issues: first, LLMs can be misled by such noise, leading to incorrect answers ( Shi et al. 2023 ; Jin et al. 2024a ; Yoran et al. 2024 ; Wu et al. 2024a ) ; second, LLMs struggle to identify and utilize key information effectively as context length increases ( Liu et al. 2024 ) ; and third, irrelevant content unnecessarily inflates input sequences, escalating computational overhead. To mitigate these issues, context compression has emerged as a promising solution to filter out irrelevant information before generation. Existing methods can be categorized into abstractive and extractive approaches. Abstractive methods leverage LLMs to summarize or rewrite retrieved content via autoregressive generation ( Xu et al. 2023a ; Yoon et al. 2024 ; Zhu et al. 2024 ) . While achieving high compression rates, they incur significant latency due to token-by-token decoding. Extractive methods instead select relevant spans from the original content, offering greater efficiency ( Jiang et al. 2024 ; Hwang et al. 2024 ; Chirkova et al. 2025 ) . However, current extractive methods typically only assess the relevance of individual sentence or document to the query, limiting their ability to integrate information across broader context. Furthermore, many such approaches rely on fixed compression rates or target lengths ( Xu et al. 2023a ; Jiang et al. 2024 ) , ignoring the variable proportion of relevant content and risking under- or over-compression. Consequently, we posit that an effective context compression method should exhibit three key properties: (1) Adaptive : It should dynamically adjust the compression rates based on the proportion of relevant information within the context. (2) Efficient : It should maintain low computational cost and latency, ensuring rapid processing for real-time applications. (3) Context-Aware : It should integrate and synthesize information from the entire retrieved content to accurately identify relevant segments. However, to the best of our knowledge, no existing compression method simultaneously satisfies all three of these properties. To bridge this gap, we introduce AttnComp (Attention-guided Context Compression), an adaptive, efficient and context-aware extractive compression method that leverages the inherent attention mechanisms of LLMs. As illustrated in Figure 1 , the AttnComp pipeline consists of two stages: (1) Attention Computation. Given a prompt that combines the instruction, retrieved documents and query, we compute attention weights from middle layers of the LLM to quantify the relevance of each text segment to the query. (see Sec. 4.1 for details). (2) Top-P Compression. We aggregate the attention weights to compute scores for the instruction and each document. Documents are then ranked by score, and the top ones are retained until their cumulative score, combined with that of the instruction, reaches a predefined threshold. Compared to fixed-length compression, this approach adjusts the retained content based on attention distribution, allowing for flexible selection ranging from no documents to all documents (see Sec. 4.2 for details). We observe that while the attention mechanisms in LLMs inherently capture relevance, they can still assign high attention to irrelevant content. This is particularly evident when all retrieved documents are irrelevant, as the model fails to shift attention away from the documents, with some irrelevant ones consistently receiving high attention. To address this, we fine-tune the cross-attention layer of the model to direct attention to relevant documents when present, or to the instruction when all documents are irrelevant (see Sec. 4.3 for details). Experimental results on multiple QA datasets highlight the superior performance of AttnComp. It achieves a 1.9 point accuracy improvement over the uncompressed baseline, while other compression methods incur at least a 3 point decrease. This advantage is even more pronounced in multi-hop question answering, which requires integrating information from multiple documents and thus places higher demands on context-aware compression capabilities. Here, our method yields at least a 5.4 point improvement over other sentence-level compression methods. Beyond accuracy, AttnComp achieves a 17x compression rate, outperforming all other evaluated extractive methods, and significantly reduces the RAG system’s end-to-end latency to 49% of the uncompressed baseline. Beyond its primary role in compression, AttnComp also offers a valuable capability for estimating the confidence of RAG responses by leveraging the attention assigned to the instruction (see Sec. 4.4 for details). After training, the attention allocated to the instruction correlates with the quality of retrieved documents, serving as an indicator of answer reliability. Experiments show a strong positive correlation between the confidence score and actual answer accuracy, enabling users to assess response trustworthiness and mitigate risks from low-quality retrieval. Furthermore, this capability also suggests a possible avenue for future research on autonomous iterative RAG ( Asai et al. 2023 ; Su et al. 2024 ; Yu et al. 2024 ) . In summary, our contributions are as follows: 1. We propose AttnComp, a novel extractive compression framework for RAG that is adaptive, efficient, and context-aware. 2. Our method enables confidence estimation for RAG responses, allowing users assess reliability and mitigate risks from low-quality retrieval. 3. Extensive experiments show that AttnComp outperforms existing compression methods and uncompressed retrieval baselines, delivering higher accuracy and lower end-to-end latency. 2 Related Work Context Compression. The compress methods can be broadly categorized into abstractive and extractive