BEE-RAG: Balanced Entropy Engineering for Retrieval-Augmented Generation
Yuhao Wang, Ruiyang Ren, Yucheng Wang, Jing Liu, Wayne Xin Zhao, Haifeng Wang · 2025
Read BEE-RAG: Balanced Entropy Engineering for Retrieval-Augmented Generation by Yuhao Wang, Ruiyang Ren, Yucheng Wang, Jing Liu, Wayne Xin Zhao, Haifeng Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-05100.txt.
Introduction
Yuhao Wang and co-authors wrote a 2025 paper on retrieval-augmented generation, where a language model reads long sets of fetched text before answering. Long context brings a problem: attention spreads thin and a measure called entropy grows without check. Their frame, BEE-RAG, aims to hold that measure steady as length changes.
By reworking how attention scores are set, they free sharpness of focus from context length. A zero-shot rule weighs which fetched parts matter most. A light tuning step then finds the best balance for each setup. Tests across many retrieval tasks show clear gains. The authors frame long context as a balance problem, not just a size problem.
Catalog record
| CREATOR | Yuhao Wang, Ruiyang Ren, Yucheng Wang, Jing Liu, Wayne Xin Zhao, Haifeng Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-05100
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BEE-RAG: Balanced Entropy Engineering for Retrieval-Augmented Generation Yuhao Wang, Ruiyang Ren, Yucheng Wang, Jing Liu, Wayne Xin Zhao, Haifeng Wang Abstract With the rapid advancement of large language models (LLMs), retrieval-augmented generation (RAG) has emerged as a critical approach to supplement the inherent knowledge limitations of LLMs. However, due to the typically large volume of retrieved information, RAG tends to operate with extremely long context lengths. From the perspective of entropy engineering, we identify unconstrained entropy growth and attention dilution due to long retrieval context as significant factors affecting RAG performance. In this paper, we propose the balanced entropy-engineered RAG (BEE-RAG) framework, which improves the adaptability of RAG systems to varying context lengths through the principle of entropy invariance. By leveraging balanced context entropy to reformulate attention dynamics, BEE-RAG separates attention sensitivity from context length, ensuring a stable entropy level. Building upon this, we introduce a zero-shot inference strategy for multi-importance estimation and a parameter-efficient adaptive fine-tuning mechanism to obtain the optimal balancing factor for different settings. Extensive experiments across multiple RAG tasks demonstrate the effectiveness of BEE-RAG. Introduction Retrieval-augmented generation (RAG) has emerged as a transformative paradigm for augmenting large language models (LLMs) with dynamically integrated external knowledge ( Gao et al. 2023 ; Lewis et al. 2020 ) . While conventional RAG systems demonstrate remarkable capabilities in knowledge-intensive tasks, their performance exhibits critical fragility when processing extended contextual inputs ( Ren et al. 2023 ) . This vulnerability arises from the inherent requirement of RAG frameworks to incorporate external retrieved documents, which typically results in substantially longer input sequences compared to standard generation tasks ( Zhang et al. 2024a ) . This limitation manifests acutely in scenarios demanding long-context comprehension, such as scientific literature analysis, multi-hop reasoning, and cross-document synthesis, where the growing complexity of modern knowledge systems necessitates robust processing of extended text spans ( Liu et al. 2023 ) . Figure 1: In vanilla RAG, increasing input context length raises the information entropy of attention scores (left), potentially harming performance, while the LLM is less focused on important segments (right). This paper proposes a balanced entropy engineering strategy, which maintains entropy stability for longer contexts and guide the LLM to focus on critical segments. Typically, existing efforts to mitigate this challenge focus predominantly on heuristic document filtering, extrapolative training, or architectural modifications, introducing worth noting trade-offs. Threshold-based retrieval document truncation or selection reduces context length at the cost of discarding potentially relevant information ( Jeong et al. 2024 ; Wang et al. 2024 ) . While extended context training may enhance RAG performance ( Lin et al. 2024 ; Zhang et al. 2024a ) , it substantially increases computational demands and risks compromising model generalization. The introduction of auxiliary modules, such as specialized encoders ( Yen, Gao, and Chen 2024 ) , further escalates the complexity of the system. As shown in Figure 1 , we identify a fundamental oversight in existing explorations: the failure to address the core tension between context entropy growth and attention allocation dynamics. As retrieved document length increases, unconstrained context entropy expansion progressively dilutes attention distributions, undermining LLMs’ capacity to prioritize semantically critical content. Such a phenomenon substantially degrades RAG’s ability to extract and utilize salient information. In this work, we fundamentally rethink RAG’s context length adaptability through the perspective of entropy engineering, and propose Balanced Entropy-Engineered RAG ( BEE-RAG ), a principled framework that establishes entropy-invariant dynamics to decouple attention sharpness from context length. First, we introduce balanced context entropy, a novel attention reformulation incorporating document-specific balancing entropy factor \beta to enforce entropy invariance across variable context lengths. Theoretical analysis demonstrates this mechanism maintains entropy levels within a stable regime for various context scales, contrasting with conventional entropy scaling. Building on this foundation, we develop intrinsic multi-importance inference, a zero-shot strategy that derives balancing entropy factors through context aggregation while preserving inter-document independence via parallel context scoring and generation. To ensure broad applicability, we further propose adaptive balancing factor learning, which is a parameter-efficient tuning method that learns task-specific balancing factor through lightweight linear projections for domain adaptation with minimal parameter updates (0.014% of total parameters). Empirical analyses across multiple real-world benchmarks reveal that BEE-RAG fundamentally alters the entropy scaling laws governing conventional RAG systems, which provides a robust and scalable solution for RAG systems dealing with long-context scenarios. Our main contributions are summarized as follows: • We introduce the concept of balanced context entropy, a novel attention reformulation that ensures entropy invariance across varying context lengths, and allocates attention to important segments. It addresses the critical challenge of context expansion in RAG. • We propose a zero-shot strategy called intrinsic multi-importance inference and parametric-efficient fine-tuning strategy adaptive balancing entropy factor learning to obtain the balancing entropy factor in different scenarios while maintaining efficiency. • Extensive empirical validation across multiple benchmarks demonstrates the effectiveness and efficiency of BEE-RAG. Preliminaries Task Formulation. The task of retrieval-augmented generation (RAG) involves the integration of retrieval-based techniques with generative models to enhance the accuracy and coherence of text generation tasks ( Lewis et al. 2020 ) . A typical RAG system consists of a corpus C , a retriever R , and a generative model LLM . Given a user query or input context q , the retriever R retrieves relevant documents D_{\text{q}} from the corpus. The generative model LLM uses both the query q and the retrieved documents to generate a response: a=\arg\max_{y}P(y\mid LLM(q,D_{q})). (1) We focus on how to enhance the multi-document utilization ability of the language model given D_{q} and q , with an emphasis on improving the LLM’s ability to leverage contextual knowledge, thereby improving its performance on factual generation. Entropy Engineering. In this work, we reconsider RAG from an information-theoretic perspective and introduce the concept of Entropy Engineering . In the standard Transformer ( Waswani et al. 2017 ) , the attention is computed as: \text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\text{Softmax}\left(\frac{\mathbf{QK}^{T}}{\sqrt{d}}\right)\mathbf{V}, (2) where \mathbf{Q} , \mathbf{K} , \mathbf{V}\in\mathbb{R}^{n\times d} represent the query, key, and value matrices, respectively, and the scaling factor \sqrt{d} stabilizes the dot-product magnitudes. This can be simplified by introducing a scaling parameter \lambda=\frac{1}{\sqrt{d}} : a_{i,j}=\frac{e^{\lambda\,\bm{q_{i}}\cdot\bm{k_{j}}}}{\sum_{j=1}^{n}e^{\lambda\,\bm{q_{i}}\cdot\bm{k_{j}}}}. (3) We examine this formula from two perspectives: From the perspective of token t_{j} being attended to: a larger a_{i,j} indicates that the LLM pays more attention to the information at position i . Therefore, the LLM should be guided to focus on relatively important information. From the perspective of token t_{i} receiving information, the values of \{a_{i,j}\ |i\leq j\} across all positions collectively determine how much information t_{i} receives. Based on this, inspired by existing analyses ( Zhang et al. 2024c ) , we measure the amount of information a position receives using discrete entropy, as shown in the following equation: H_{i}=-\sum_{j=1}^{n}a_{i,j}\log a_{i,j}, (4) which quantifies how much information t_{i} receives from the attention perspective. This insight suggests that LLMs struggle with longer sequences when not trained on them, likely due to the discrepancy in information received by tokens in longer contexts. Based on the previous analysis, the optimization of attention entropy should focus on two aspects: • The information entropy at positions that are relatively important and likely contain key information should increase. • The overall total information entropy of the context should ideally