Generating Leakage-Free Benchmarks for Robust RAG Evaluation
Jiayi Liu*, Jiaxing Zhang, Bowen Jin, Jennifer Neville* · 2026
Read Generating Leakage-Free Benchmarks for Robust RAG Evaluation by Jiayi Liu*, Jiaxing Zhang, Bowen Jin, Jennifer Neville* (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-08838.txt.
Introduction
Jiayi Liu, Jiaxing Zhang, Bowen Jin, and Jennifer Neville build benchmarks for retrieval augmented generation that models cannot pass from memory alone. Many current test sets leak: their questions are answerable from a model's built in knowledge, and reuse for training makes the leak worse over time. Their SeedRG pipeline extracts a reasoning graph from a seed set's question and context pairs, then swaps entities under type rules to mint fresh, structurally similar tasks.
Two checks guard quality: a consistency check keeps difficulty honest, and a leakage filter drops anything solvable without retrieval. Evaluation of retrieval systems gains tests that stay novel as models absorb older benchmarks, keeping the measurement aimed at retrieval itself.
Catalog record
| CREATOR | Jiayi Liu*, Jiaxing Zhang, Bowen Jin, Jennifer Neville* |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.08838
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generating Leakage-Free Benchmarks for Robust RAG Evaluation Jiayi Liu*, Jiaxing Zhang, Bowen Jin, Jennifer Neville* Abstract Retrieval-augmented generation (RAG) is widely used to augment large language models (LLMs) with external knowledge. However, many benchmark datasets, designed to test RAG performance, comprise many questions that can already be answered from an LLM’s parametric memory. This leads to unreliable evaluation. We refer to this phenomenon as knowledge leakage—cases where RAG tasks are solvable without retrieval. This issue worsens over time due to benchmark aging. As benchmarks are reused for training, their contents are increasingly absorbed into model parameters, making them less effective for evaluating retrieval. We introduce SeedRG , a semi-synthetic benchmark generation pipeline that mitigates knowledge leakage and addresses the issue of benchmark aging. Starting from a seed benchmark dataset, SeedRG extracts a reasoning graph from question–context pairs to capture their underlying reasoning structure, and then generates new examples via type-constrained entity replacement. This process produces structurally similar but novel instances that are unlikely to exist in the model’s parametric knowledge, while preserving the original reasoning patterns. To ensure quality, we incorporate two verification steps: (1) a reasoning-graph consistency check to maintain task difficulty, and (2) a knowledge-leakage filter to exclude instances answerable without retrieval. We evaluate SeedRG on three seed benchmarks (HotpotQA, 2WikiMultihopQA, QASC) and three popular LLMs (GPT-5, Claude Sonnet 4.5, Gemini 2.5 Flash). SeedRG reduces knowledge leakage by at least 78% while preserving reasoning difficulty. By removing the confounding effect of parametric knowledge, SeedRG reveals meaningful variability across RAG systems that is otherwise obscured. Prior benchmarks show uniformly high performance across RAG methods (HippoRAG, GraphRAG, OGRAG, SemanticRAG), because performance is dominated by model knowledge. In contrast, SeedRG surfaces clear differences in retrieval and reasoning ability across the methods. Beyond benchmark construction, we provide a systematic analysis linking reasoning difficulty to graph structure, showing how structural variations induce predictable changes in model accuracy. Together, these results demonstrate that SeedRG enables more discriminative and robust evaluation of RAG systems. 1 Introduction Retrieval-Augmented Generation (RAG) Lewis et al. 2020 is widely used to improve large language models (LLMs) by incorporating external knowledge at inference time. Because RAG avoids retraining by leveraging retrieval, it has become a standard approach for knowledge-intensive tasks. Progress on RAG systems is typically measured using multi-hop question answering benchmarks, which are intended to require retrieval over multiple documents. However, these benchmarks often fail to test retrieval at all. We find that on widely used multi-hop QA datasets ( ) , LLMs can answer even more than half of questions correctly without retrieving any documents . When the majority of questions are solvable from LLM parametric memory alone, evaluation results are dominated by the model’s internal knowledge rather than the quality of retrieval. As a result, current benchmarks cannot meaningfully distinguish RAG systems—even weak retrievers inherit strong performance from the underlying model. This issue undermines a large body of empirical work. RAG systems are now central to applications such as open-domain question answering Petroni et al. 2021 , fact verification, and domain-specific reasoning, with a growing ecosystem of retrieval methods—including dense, graph-based, and ontology-guided approaches—competing on shared benchmarks. When benchmarks can be solved from parametric memory, performance is dominated by the underlying model rather than the retriever, collapsing the observable differences between systems. This masks variation in retrieval quality and limits our ability to measure how different retrieval strategies influence downstream reasoning. We identify two mechanisms that drive this evaluation failure. Knowledge leakage Agarwal et al. 2024 ; Wu et al. 2025 ; Desai et al. 2026 ; Yoon et al. 2025 occurs when benchmark questions are answerable from an LLM’s parametric memory, making retrieval unnecessary. Benchmark aging Zhou et al. 2023 ; Zhang et al. 2026 compounds this effect over time. As benchmarks are reused in training and data curation pipelines, their contents are absorbed into model parameters, progressively reducing their ability to test retrieval. Together, these mechanisms collapse the evaluation signal, obscuring differences between systems. Addressing this problem requires benchmarks that are (1) outside the model’s parametric knowledge and (2) regenerable to remain robust to benchmark aging. A natural approach is to generate new data using LLMs. However, naive generation fails to meet these requirements: generated questions often reuse well-known entities (leading to continued leakage), may introduce factual errors in supporting context, and provide no control over reasoning difficulty. Figure 1: An example of the RAG evaluation gap. (a) Existing benchmarks overlap with LLM pretraining data, making retrieval redundant. (b) SeedRG preserves the reasoning structure of seed questions while replacing all entities with novel counterparts, ensuring no overlap with parametric knowledge. We introduce SeedRG , a semi-synthetic benchmark generation pipeline that addresses these challenges. Starting from a seed benchmark, SeedRG extracts a reasoning graph from each question–context pair to capture its underlying structure, and then generates new examples via type-constrained entity replacement . This produces structurally equivalent but novel instances that are unlikely to exist in the model’s parametric knowledge, while preserving the original reasoning patterns. To ensure quality, SeedRG incorporates two verification steps. A reasoning graph consistency check ensures that the transformed examples preserve the original reasoning structure and difficulty. A knowledge leakage check filters out instances that can be answered without retrieval. Together, these steps ensure that generated examples are both retrieval-dependent and difficulty-preserving. We further introduce two metrics— leakage error and answerability accuracy —to quantify the effectiveness of RAG benchmarks. In summary, our contributions are as follows: 1. We provide systematic evidence that three widely used multi-hop QA benchmarks suffer from knowledge leakage , and formalize benchmark quality in terms of leakage error and answerability accuracy. 2. We propose SeedRG , a semi-synthetic pipeline that generates leakage-free benchmarks by combining reasoning graph extraction with type-constrained entity replacement, along with dual verification to preserve difficulty and enforce retrieval dependence. 3. We show that SeedRG produces benchmarks that reduce knowledge leakage, preserve reasoning difficulty, and reveal meaningful performance differences across RAG systems that are obscured in existing benchmarks. Compared to direct LLM generation, SeedRG yields higher-quality benchmarks with substantially lower leakage and fewer factual inconsistencies. 2 Background 2.1 Synthetic datasets Synthetic Benchmarks for RAG Recent frameworks have standardized the evaluation of RAG systems through automated data synthesis. RAGEval Zhu et al. 2024 generates schema-driven datasets to assess scenario-specific factual accuracy. It defines dataset quality using three key metrics: completeness of the answer, absence of hallucination , and irrelevance of non-essential content. To scale this approach, BenchmarkQED Research 2025 employs the AutoQ tool to generate synthetic queries across a principled 2\times 2 taxonomy. It measures quality via coverage (diversity of query types) and rigor (stability of comparative rankings). However, both frameworks share a critical limitation: they fail to add restrictions that avoid generating queries already present in the LLM’s knowledge. By not explicitly disentangling parametric memory from retrieval necessity, these benchmarks struggle to isolate the true utility of the retrieval component. Synthetic Data for Instruction Tuning In the broader context of model alignment, synthetic data quality is often defined by downstream efficiency rather than retrieval isolation. Distilling Step-by-Step Hsieh et al. 2023 and Orca Mukherjee et al. 2023 demonstrate that ”good” synthetic data allows smaller student models to achieve teacher-parity with significantly fewer training samples. Similarly,