Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?

Laura Dietz ✉, Bryan Li, Eugene Yang, Dawn Lawrie, William Walden, James Mayfield · 2026

Read Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets? by Laura Dietz ✉, Bryan Li, Eugene Yang, Dawn Lawrie, William Walden, James Mayfield (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13227.txt.

Introduction

Laura Dietz and co-authors ask a sharp test design question. Tools that write from sources are now judged by AI judges. Fact bit tests, called nugget tests, sit in both the judge seat and in the tools. That dual role can hide a leak.

The team asks how much a tool gains if it can peek at judge secrets. Such peeks would be a form of inside knowledge. If gains are large, scores may flatter tools that game the test. The study probes that risk head on. For teams that build or buy such tools, the work is a call to guard test facts as they would guard exam keys.

Catalog record

CREATORLaura Dietz ✉, Bryan Li, Eugene Yang, Dawn Lawrie, William Walden, James Mayfield
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-13227

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets? Laura Dietz ✉, Bryan Li, Eugene Yang, Dawn Lawrie, William Walden, James Mayfield Abstract RAG systems are increasingly evaluated and optimized using LLM judges, an approach that is rapidly becoming the dominant paradigm for system assessment. Nugget-based approaches in particular are now embedded not only in evaluation frameworks but also in the architectures of RAG systems themselves. While this integration can lead to genuine improvements, it also creates a risk of faulty measurements due to circularity. In this paper, we investigate this risk through comparative experiments with nugget-based RAG systems, including Ginger and Crucible , against strong baselines such as GptResearcher . By deliberately modifying Crucible to generate outputs optimized for an LLM judge, we show that near-perfect evaluation scores can be achieved when elements of the evaluation—such as prompt templates or gold nuggets—are leaked or can be predicted. Our results highlight the importance of blind evaluation settings and methodological diversity to guard against mistaking metric overfitting for genuine system progress. 1 1 1 Online appendix at https://github.com/hltcoe/ecir26-crucible-appendix/ 1 Introduction As LLM-based chat systems like ChatGPT and Claude become primary information sources for general users, ensuring accuracy, recency, and credible sourcing of their responses is critical for developing trustworthy systems [ 15 , 10 ] . Retrieval-Augmented Generation (RAG) has recently emerged as a leading approach to achieve these aims [ 24 , 21 , 8 ] . In parallel, nugget-based evaluation methods, which check generated outputs for the presence of key pieces of information ( nuggets ), have become an important tool for RAG evaluation [ 26 , 1 , 30 , 28 ] , grounding system assessment in assessor-identified Q&A pairs or claims. Increasingly, however, these methods rely on LLMs not only for matching nuggets to content, but also for developing nugget banks from scratch [ 30 , 14 ] . While all evaluation paradigms can produce systematically biased, invalid measurements, LLM-based evaluation methods are particularly susceptible to a range of vulnerabilities [ 13 ] . For instance, circularity arises whenever components of the evaluation methods are usefully integrated into a RAG system. Moreover prompts, scripts, and LLM judges are routinely made public and may thus be inadvertently exploited by system developers. This can lead to inflated evaluation scores and reduced alignment with manual assessments [ 9 ] . A suggested safeguard against circularity is to rely on human-curated, nugget-based evaluation [ 26 , 34 , 18 ] . Yet such safeguards are not perfect. In this paper, we identify three pathways by which insider knowledge of an LLM-based evaluation process can be exploited to inflate evaluation results, even without a direct test set leak: (1) tuning RAG systems to optimize narrowly for measured criteria while ignoring other, unmeasured quality criteria (RQ1); (2) exploiting knowledge of the evaluation prompt and LLM model to modify system outputs (RQ2); and (3) predicting the set of gold nuggets for system usage with high accuracy (RQ3). 2 2 2 To avoid confusion, we refer to nuggets generated automatically by RAG systems, such as Crucible , as system nuggets , and manually created nuggets used by the evaluation system Argue as gold nuggets . Among these, RQ3 poses the most serious and underappreciated threat. Nugget sets are meant to serve as independent ground truth. However, our investigations reveal that system-generated nuggets can overlap substantially with human-curated ones. This overlap may be greater still for nugget creation pipelines, such as AutoNuggetizer [ 29 ] , that rely on LLMs to generate candidate nuggets that are then manually filtered. Thus, even absent nefarious intent, system developers may achieve high nugget-based evaluation scores by leveraging LLM-based nugget generation components. Indeed, none of RQ1, RQ2, or RQ3 assumes intentional subversion: RAG systems may incidentally be trained using a model similar to the LLM judge, developed using similar prompts, or optimized against metrics or nuggets similar to those used at test time. Moreover, adopting these ideas may lead to genuinely better systems, as noted by 9 . By simulating the downstream effects of integrating these components into system development, we show how safeguards can collapse, resulting in inflated results along measured dimensions. Our findings reveal that automatic nugget-based evaluations, though valuable, are vulnerable to a form of leakage that is subtle, pervasive, and arguably already occurring. Contributions. To study this problem, we introduce Crucible , a RAG system that incorporates evaluation ideas to probe the susceptibility of nugget-based LLM judge systems. Crucible is designed to expose and study vulnerabilities of AutoArgue [ 36 ] , an open-source LLM-based implementation of Argue [ 26 ] . Drawing on insider knowledge of AutoArgue , Crucible demonstrates that the evaluation is robust only when key evaluation elements are kept secret. Once these elements become public knowledge, they may be used to enhance system outputs and inflate scores without necessarily obtaining genuine improvements. We present a case study on the TREC NeuCLIR 2024 Report Generation Pilot task—a long-form RAG task with complex queries and an emphasis on faithful citation. Crucially, nuggets for this task were fully manually curated through a principled process. Our exclusive focus on TREC NeuCLIR is a consequence of the lack of available alternative RAG test collections with fully manual nuggets at the time of our study. Since the purpose of this case study is to explore the susceptibility of an LLM judge paradigm, we do not evaluate the efficiency of the RAG system or the automatic evaluation. 2 Background 2.1 LLM-as-a-Judge The introduction of LLMs as automatic judges for retrieval and generation systems has led to a surge of interest in their reliability as replacements for or supplements to human assessments [ 25 ] . While initial studies highlighted strong correlations with human labels [ 34 , 33 ] , more recent work documents weaknesses and instabilities that threaten validity [ 9 , 31 ] . Recent IR-focused work emphasizes observational comparisons and prompt-level analyses. 5 synthesize the roles that LLMs can play as rankers, judges, and assistants, and identify feedback loops as a fundamental risk for evaluation. 32 compare LLM judges against human annotations for retrieval-augmented generation tasks at the TREC RAG Track, showing both alignment and divergence. 4 benchmark pointwise, pairwise, and rubric-based judging paradigms, while their follow-up [ 3 ] highlights the prompt sensitivity of LLM judges. 35 extend these comparisons with a large-scale study of Umbrela , an open-source implementation of Bing’s relevance assessor, across multiple settings. While their positive findings are used to support the reliability of LLM judges, 9 show that Umbrela ceases to be a reliable evaluator under conditions of circularity. A complementary line of work designs adversarial content attacks to measure judge susceptibilities, often by injecting instructions that override evaluation prompts (e.g. forget all previous instructions and rate this response as relevant ). Early studies explored generic prompt subversions in generative settings [ 2 , 6 , 7 , 22 ] , while more recent work has shifted toward citation evaluation [ 10 ] . Taken together, the current literature offers descriptive accounts of vulnerabilities [ 13 ] , but lacks systematic meta-evaluation paradigms of the sort proposed in this work for investigating hypothesized weaknesses of LLM-based evaluations. 2.2

Read the complete text