SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression

Yiqiao Jin, Kartik Sharma, Vineeth Rakesh, Yingtong Dou, Menghai Pan, Mahashweta Das, Srijan Kumar · 2025

Read SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression by Yiqiao Jin, Kartik Sharma, Vineeth Rakesh, Yingtong Dou, Menghai Pan, Mahashweta Das, Srijan Kumar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05633.txt.

Introduction

Yiqiao Jin and six co-authors improve how AI systems use fetched documents in a 2025 paper. Retrieval systems feed outside text to a language model before it answers. Retrieved pages can be long and full of repeats. Trimming them risks cutting facts the answer needs.

Their frame, SARA, keeps evidence in two forms at once. Short plain text spans hold names, numbers, and other sharp details. Compact vectors carry the broad sense of longer passages. A selection loop reranks evidence using those vectors as it goes. Tests ran on nine data sets with five open models. Scores for relevance and correctness rose by clear margins. The authors argue both forms are needed for tight context budgets.

Catalog record

CREATORYiqiao Jin, Kartik Sharma, Vineeth Rakesh, Yingtong Dou, Menghai Pan, Mahashweta Das, Srijan Kumar
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-05633

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SARA: Selective and Adaptive Retrieval-augmented Generation with Context Compression Yiqiao Jin, Kartik Sharma, Vineeth Rakesh, Yingtong Dou, Menghai Pan, Mahashweta Das, Srijan Kumar Abstract Retrieval - augmented Generation (RAG) extends large language models (LLMs) with external knowledge but faces key challenges: restricted effective context length and redundancy in retrieved documents. Pure compression-based approaches reduce input size but often discard fine-grained details essential for factual accuracy. We propose SARA, a unified RAG framework that balances local precision and global knowledge coverage under tight context budgets. SARA combines natural-language text snippets with semantic compression vectors to jointly enhance context efficiency and answer correctness. It represents contexts at two complementary levels: 1) fine-grained natural-language spans that preserve critical entities and numerical values, and 2) compact , interpretable vectors that summarize high-level semantics. An iterative evidence-selection module employs the compression vectors for dynamic reranking of contexts. Across 9 datasets and 5 open-source LLMs spanning 3 model families (Mistral, Llama, and Gemma), SARA consistently improves answer relevance (+17.71), answer correctness (+13.72), and semantic similarity (+15.53), demonstrating the importance of integrating textual and compressed representations for robust, context-efficient RAG. 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities across various natural language understanding and generation tasks ( Xiao et al. 2024a ; Zhao et al. 2024 ) . Meanwhile, as LLMs are parametric in nature, their knowledge is inherently constrained by the scope, domain, and recency of their training data ( Jin et al. 2024b ; Liu et al. 2025 ) . Retrieval-augmented generation (RAG) ( Lewis et al. 2020 ) addresses this by retrieving from external non-parametric knowledge sources, essential for knowledge-intensive tasks. Challenges. Despite its promise, RAG still faces key challenges in effectively retrieving, selecting, and integrating external evidence. 1) Limited Effective Context. While some LLMs support long inputs, their attention is biased toward earlier tokens ( Li et al. 2024b ) , making them sensitive to input order and prone to overlooking important information near the end of the input ( Yu et al. 2024 ) . Extending usable context often requires costly, model-specific architectural changes ( Ding et al. 2023 ) . 2) Context Redundancy. Retrieved documents often include redundant or loosely structured content (e.g. transcripts or news articles) ( Yu et al. 2024 ; Ge et al. 2024 ) . Without careful post-processing, duplicate or irrelevant content inflates token usage, distracts the model, degrades answer quality or even leads to hallucinations. 3) Compression-fidelity Trade-off. Existing context compression techniques reduces input length but often sacrifice fine-grained details (e.g. numeric values, organization names, and geographical locations), leading to hallucinated or incomplete responses. While existing methods achieve high compression rates, aggressive compression process risk discarding critical information essential for factual accuracy. This Work. We present SARA, a unified RAG framework that improves both retrieval and generation stages through structured evidence compression and adaptive selection. From the generation perspective, SARA represents long contexts using a small number of semantically rich, self-contained compression vectors , which act as lightweight abstractive summaries that preserve essential information while significantly reducing input length. Specifically, we leverage state-of-the-art embedding models ( Meng et al. 2024 ; Muennighoff et al. 2023 ) to encode retrieved documents into multiple, semantically rich compression vectors. These vectors are also explainable and can be interpreted through auto-encoding to reveal their underlying semantics. From the retrieval perspective, SARA introduces an iterative evidence selection mechanism that leverages the compression vectors to dynamically refine the set of top-ranked documents. SARA progressively selects contexts based on the knowledge required to properly address the query and knowledge coverage of existing contexts, minimizing redundancy while maximizing informativeness. SARA is agnostic to the choice of embedding models, open-source LLMs, and retrievers. Our contributions are as follows: • We propose SARA, a novel RAG framework for long-context tasks. SARA introduces a hybrid compression strategy , balancing local precision using natural language spans and global abstraction via compression vectors, enabling fine-grained reasoning and holistic understanding within strict context budgets. • We propose an iterative context refinement mechanism based on the compression vectors to dynamically optimize the retrieved context by reducing redundancy and prioritizing query-relevant content. • Comprehensive experiments on 5 LLMs spanning 3 model families, including Mistral-7B, MistralNemo-12B, MistralSmall-24B, Llama-3.1-8B, and Gemma3-4B, demonstrate that SARA consistently improves performance and generalizes well across LLMs (Section 3.3 ), retrievers (Section 3.4 ), and embedding models (Appendix B.2 ). 2 Method 2.1 Problem Formulation A retrieval-augmented generation (RAG) pipeline consists of a retriever that fetches relevant evidence from a large-scale corpus based on the input query and a generator that synthesizes the evidence to answer the query. Given a query q and corpus C , the retriever \mathcal{R}(\cdot) selects the top- n relevant contexts \mathcal{V}_{\mathrm{sel}}\subseteq C . To improve effectiveness, RAG may incorporate a reranking step to reorder the input documents, prioritizing the most relevant ones for answer generation. 2.2 Overview LLMs have limited effective context windows, and performance degrades when key information is buried in long inputs ( Jiang et al. 2024 ) . SARA mitigates this by compressing long context into compact vectors while selectively retaining essential evidence in natural language, preserving model capacity for the most relevant content. SARA follows a two-stage training procedure: During Compression Learning , SARA learns to reconstruct original context from compression vectors. In Instruction-tuning , SARA is adapted to rerank the evidence using the compression vectors and reason over mixed inputs–combining natural language and compressed evidence. Our method is model-agnostic , compatible with any retrievers, embedding models, and open-source LLMs. A lightweight projection layer aligns the embedding space with the LLM space, requiring no significant changes to internal components like the attention mechanism, enabling seamless integration with future embedding models and LLMs. Sample prompts for all stages are provided in Table 9 . 2.3 Compression Learning Figure 1 : SARA reasons over a mixture of compressed evidence and natural language contexts to balance local precision and global coverage when generating responses. An iterative evidence reranking step selects contexts for relevance and diversity. The retriever, compressor, and QA model uses a variety of embedding models. An effective compression mechanism should meet three core principles: 1) Semantic Fidelity –preserving sufficient information for accurate context reconstruction; 2) Token Compatibility –producing compression vectors interpretable by LLMs via prompting; and 3) Scalability –requiring minimal adaptation across retrievers and LLMs. To meet these goals, SARA leverages sentence embeddings ( Reimers and Gurevych 2019 ) aligned with the LLM’s token space, enabling compact and interpretable representations that support reasoning under tight context budgets. Embedding Alignment. SARA encodes each text chunk into a compression vector that fits within a single token’s embedding space (Figure 6 ). A lightweight compressor–combining a sentence embedding model and an MLP–is trained via an autoencoding task ( Liu et al. 2023 ; Cheng et al. 2024 ) to align sentence embeddings with the LLM’s token space: \mathcal{L}_{\mathrm{align}}(s_{i})=-\log P_{\theta}\left(s_{i}\mid\mathrm{Enc}(s_{i}),x_{\text{ins}}\right). (1) Here, s_{i} is a text chunk, \mathrm{Enc}(\cdot) is the compressor, \theta is the model’s parameter, and x_{\text{ins}} is the decoding instruction such as “The token <C> can be interpreted as: [CHUNK] .” As one compression vector has limited representation capacity, we segment each document into chunks, and encode each chunk as a separate compression vector. We adopt a curriculum learning strategy (

Read the complete text