Scaling DPPs for RAG: Density Meets Diversity
Xun Sun, Baiheng Xie, Li Huang, Qiang Gao · 2026
Read Scaling DPPs for RAG: Density Meets Diversity by Xun Sun, Baiheng Xie, Li Huang, Qiang Gao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03240.txt.
Introduction
Sun and colleagues improve retrieval for grounded generation by picking evidence for richness and range together, not relevance alone. Their ScalDPP method uses determinantal point processes through a light module to model links between chunks and pick matching context. A new set-level loss forces true matching evidence chains to beat equally sized redundant rivals under that geometry.
Catalog record
| CREATOR | Xun Sun, Baiheng Xie, Li Huang, Qiang Gao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53049 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.03240
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scaling DPPs for RAG: Density Meets Diversity Xun Sun, Baiheng Xie, Li Huang, Qiang Gao Abstract Retrieval-Augmented Generation (RAG) enhances Large Language Models (LLMs) by grounding generation in external knowledge, yielding relevance responses that are aligned with factual evidence and evolving corpora. Standard RAG pipelines construct context through relevance ranking, performing point-wise scoring between the user query and each corpora chunk. This formulation, however, ignores interactions among retrieved candidates, leading to redundant contexts that dilute density and fail to surface complementary evidence. We argue that effective retrieval should optimize jointly for both density and diversity, ensuring the grounding evidence that is dense in information yet diverse in coverage . In this study, we propose ScalDPP, a diversity-aware retrieval mechanism for RAG that incorporates Determinantal Point Processes (DPPs) through a lightweight P-Adapter, enabling scalable modeling of inter-chunk dependencies and complementary context selection. In addition, we develop a novel set-level objective, Diverse Margin Loss (DML), that enforces ground-truth complementary evidence chains to dominate any equally sized redundant alternatives under DPP geometry. Experimental results demonstrate the superiority of ScalDPP, substantiating our core statement in practice. 1 Introduction Figure 1 : Standard RAG models query-chunk relevance but neglects inter-chunk diversity and complementarity. Example from MultiHop-RAG ( 24 ) . Large language models (LLMs) have achieved strong performance across a wide range of natural language understanding and generation tasks. Nevertheless, they remain fundamentally constrained by the probabilistic autoregressive nature, which prioritizes textual coherence over factual accuracy, leading to inappropriate outputs, such as hallucinated or outdated content ( 25 ) . Retrieval-Augmented Generation (RAG) mitigates these limitations by dynamically retrieving and incorporating external, domain-specific knowledge during the generation process ( 19 ; 11 ) , enabling relevance-aware responses that are better grounded in factual information. In standard RAG pipelines, the corpus is partitioned into fixed-length textual chunks, from which a small set of relevant candidates is retrieved via embedding-based similarity matching with the input query ( 9 ; 8 ) . These candidates are then concatenated and provided to LLMs as additional context, optionally followed by a reranking stage that refines relevance scores. Such pipelines implicitly assume that the top-ranked chunks are sufficient. However, these candidates are selected based on their similarity to the input query, which inevitably produces clusters of near-duplicate chunks, such as multiple paraphrases of the same fact. Under a limited context window, such redundancy dilutes the effective token budget and constrains the generator to reason over a narrow semantic slice ( 14 ; 27 ; 28 ) . Moreover, similarity-driven retrieval further overlooks chunks that are individually weaker matches but collectively essential, as it fails to account for orthogonal attributes, latent constraints, or cross-cutting perspectives required for multi-hop reasoning. Thus, what appears as evidence can be misleading: redundant chunks crowd out uniquely informative context. As a result, candidates are highly correlated with the query, while inter-candidate interactions – particularly diversity and complementarity – are insufficiently captured, as illustrated in Fig. 1 . Although recent approaches leverage knowledge graphs to model entity-level interactions through structured relational paths ( 6 ; 10 ) , they typically depend on costly graph pre-construction. Furthermore, they emphasize explicit entity links rather than probabilistic optimization over chunk-level subsets, limiting scalability and flexibility in RAG settings ( 21 ) . Motivated by these analyses, we ask whether retrieval in RAG can be reformulated by explicitly ensuring that the grounding evidence is dense in information yet diverse in coverage . To this end, we are the first to employ Determinantal Point Processes (DPPs) ( 22 ; 12 ; 17 ) , a class of probabilistic models rooted in statistical physics and random matrix theory, into RAG systems. DPPs naturally model subset-level diversity through negative correlations, providing a principled foundation for constructing informative and complementary contexts beyond relevance-based retrieval. However, directly applying DPPs to RAG poses two major challenges. First, pre-training the kernel matrix \mathbf{L} in DPPs is computationally prohibitive, incurring \mathcal{O}(|\mathcal{D}|^{2}) storage where \mathcal{D} refers to knowledge base, thus severely restricting scalability when the knowledge base evolves through incremental updates. Moreover, since \mathbf{L} is constrained to be symmetric and positive semi-definite, DPPs can only induce negative dependencies among chunks: increasing similarity between \mathbf{D}_{i} and \mathbf{D}_{j} necessarily reduces \det(\mathbf{L}_{\mathbf{D}_{i},\mathbf{D}_{j}}) . Consequently, DPPs are limited to capturing repulsive interactions and cannot express attractive relations that are often essential. To overcome both limitations, we propose ScalDPP , a diversity-aware retrieval mechanism for RAG under DPPs geometry. ScalDPP attaches a parameter-efficient P-Adapter to the base embedding model, serving as a lightweight mapping function. During initial retrieval, P-Adapter is disabled to preserve the original query–chunk relevance, and is activated only during subset selection to inject learned inter-chunk interaction patterns into the embeddings. When inference, ScalDPP dynamically constructs the kernel \mathbf{L} over the retrieved candidate pool, optionally fusing it with a quality matrix \mathbf{Q} derived from reranker scores (otherwise \mathbf{Q}=\mathbf{I} ). Subset selection is then performed via Maximum a Posteriori (MAP) inference, yielding a context that is both diverse and complementary. In addition, we develop a novel set-level objective, Diverse Margin Loss (DML), to optimize the P-Adapter and shape the embedding space such that determinant maximization corresponds to selecting dense yet complementary contexts. Our main contributions are summarized as: 1) We introduce ScalDPP, the first plug-and-play module to extend DPP-based modeling to RAG, explicitly capturing inter-chunk diversity and complementarity beyond query–chunk relevance, 2) Unlike classical DPP formulations, we propose a scalable dynamic kernel construction mechanism coupled with an adaptive embedding adapter P-Adapter to overcome DPPs’ inherent scalability and correlation limitations, enabling complementarity-aware chunk selection, and 3) In contrast to employing standard negative log-likelihood loss (NLL), we develop a novel Diverse Margin Loss (DML) to optimize the proposed P-Adapter, with smooth surrogate formulations that ensure differentiability and favorable optimization properties. 2 Method Figure 2 : Overview of the ScalDPP approach. The pipeline integrates dynamic DPP subset selection with adaptive embeddings to achieve complementary chunk selection. An effective RAG system is ultimately governed not by how many candidates are retrieved, but by how much useful, distinct, and complementary information those chunks collectively convey. Under a fixed token budget, the generator can only take advantage of what is present in the candidate set. Thus, redundancy directly reduces the informational density of the augmentation, while homogeneity constrains unique evidence. To address this structural limitation, we frame the objective of retrieval as the task of constructing a subset whose elements are not only relevant to the query but also mutually diverse. Guided by the formulation, we propose ScalDPP , which jointly optimizes informational density, i.e., the relevance between query-candidates, and complementarity uniqueness, i.e., non-redundancy among candidates, within the available context window. The overview is illustrated in Fig. 2 . 2.1 DPP-based Subset Selection DPPs are probabilistic models for selecting diverse subsets, originating in statistical physics ( 22 ) and random matrix theory ( 12 ) . Formally, given a ground set \mathcal{Y}=\{1,2,\cdots,N\} with N chunks, a DPP defines a probability distribution over all subsets Y\subseteq\mathcal{Y} given by \small P(Y)=\frac{\det(\mathbf{L}_{Y})}{\det(\mathbf{L}+\mathbf{I})}, (1) where \mathbf{L}\in\mathbb{R}^{N\times N} is a positive semi-definite (PSD) kernel matrix capturing chunk similarities and ensuring non-negative probabilities, \mathbf{L}_{Y} denotes the submatrix of \mathbf{L} indexed by Y , and \mathbf{I} is the identity matrix. The PSD property of \mathbf{L} ensures \det(\mathbf{L}_{Y})\geq 0