Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion

Yejun Yoon Jaeyoon Jung Seunghyun Yoon Kunwoo Park · 2025

Read Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion by Yejun Yoon Jaeyoon Jung Seunghyun Yoon Kunwoo Park (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14175.txt.

Introduction

Yejun Yoon, Jaeyoon Jung, Seunghyun Yoon, and Kunwoo Park test whether a popular search trick works for the reason people think, in this 2025 arXiv paper. The trick asks a language model to invent a document that might answer a query, then searches with that text added. Scores rise, and the gain is usually credited to the model's knowledge.

Using fact checking as their test bed, the authors checked whether the invented documents quietly contained the very facts the true evidence holds. Gains clustered on claims where the invented text did echo the gold evidence. That pattern hints the test sets themselves leak answers the model memorized, flattering the method. The paper urges care in how such search gains are read.

Catalog record

CREATORYejun Yoon Jaeyoon Jung Seunghyun Yoon Kunwoo Park
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14175

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion Yejun Yoon Jaeyoon Jung Seunghyun Yoon Kunwoo Park Abstract Query expansion methods powered by large language models (LLMs) have demonstrated effectiveness in zero-shot retrieval tasks. These methods assume that LLMs can generate hypothetical documents that, when incorporated into a query vector, enhance the retrieval of real evidence. However, we challenge this assumption by investigating whether knowledge leakage in benchmarks contributes to the observed performance gains. Using fact verification as a testbed, we analyze whether the generated documents contain information entailed by ground-truth evidence and assess their impact on performance. Our findings indicate that, on average, performance improvements consistently occurred for claims whose generated documents included sentences entailed by gold evidence. This suggests that knowledge leakage may be present in fact-verification benchmarks, potentially inflating the perceived performance of LLM-based query expansion methods. 1 Introduction Zero-shot retrieval aims to identify relevant documents without requiring any relevance supervision for training a retriever Zhao et al. (2024) . Because obtaining query-document pairs, such as MS-MARCO Bajaj et al. (2016) , for supervised training is challenging, developing zero-shot retrieval methods is both difficult and highly desirable for effectively addressing knowledge-intensive applications Lewis et al. (2020) , including question answering Zhu et al. (2021) and fact verification Guo et al. (2022) . Recent studies have leveraged the natural language generation capabilities of large language models (LLMs) to enhance the performance of zero-shot retrieval Thakur et al. (2021) . LLM-based query expansion (QE) uses LLMs to generate documents that extend a query Jagerman et al. (2023) ; Lei et al. (2024) ; Mackie et al. (2023a) . Approaches such as HyDE Gao et al. (2023) and Query2doc Wang et al. (2023) , which have been widely adopted in recent research Wang et al. (2024a) ; Chen et al. (2024b) ; Yoon et al. (2024) , have achieved notable performance gains across various benchmarks without retriever parameter updates. These approaches prompt LLMs to generate documents that answer a question or verify a claim. Although these generated documents, referred to as hypothetical documents, may contain factual errors or hallucinations, it is assumed that incorporating them into a query can enhance retrieval of relevant real documents Gao et al. (2023) . Figure 1: Illustration of potential knowledge leakage in LLM-based query expansion. In this paper, we challenge the underlying assumption, as illustrated by a counterexample in Figure 1 : Do LLMs truly generate hypothetical documents, or are they merely reproducing what they already know? LLMs are extensively pretrained on vast corpora, primarily collected from the web. As common retrieval targets, such as Wikipedia and web documents, are often included in these pretraining corpora Groeneveld et al. (2024) ; Touvron et al. (2023) ; Du et al. (2022) ; Brown et al. (2020) , many available LLMs may already contain knowledge relevant to a given query and retrieval targets in benchmark datasets, a phenomenon we refer to as knowledge leakage . If knowledge leakage occurs and an LLM reproduces what it already knows about a given query, its generation cannot be seen as hypothetical documents. Furthermore, assessing QE performance on a benchmark where leakage occurs could lead to overestimation, as retrieving verbatim or semantically equivalent sentences is a much more trivial task than predicting the relevance between a query and document when the relevance signal is not explicit. To investigate whether knowledge leakage exists in popular benchmarks and how it affects downstream performance, this study uses fact verification as a testbed. We analyze whether LLM-generated documents contain sentences entailed by gold evidence and assess their impact on performance. Across experiments spanning three benchmarks and seven LLMs, we observed a consistent trend: QE methods were effective, on average, only when the generated documents included sentences entailed by gold evidence. These findings suggest that knowledge leakage may be present in these benchmarks, potentially inflating the perceived effectiveness of LLM-based QE methods. 2 Related Works LLM-based Query Expansion QE has been explored as a means to enhance retrieval performance by enriching the initial query representation Azad and Deepak (2019) . One widely studied approach is relevance feedback Rocchio (1971) ; Lavrenko and Croft (2001) ; Amati and Van Rijsbergen (2002) , which leverages feedback signals to expand the query. Recent work has explored LLM’s generative capabilities for QE Zhu et al. (2023) ; Jagerman et al. (2023) ; Lei et al. (2024) . Mackie et al. (2023b) , for instance, introduced a method that utilizes LLM-generated documents as relevance feedback. Meanwhile, other researchers proposed HyDE Gao et al. (2023) and Query2doc Wang et al. (2023) , which employs LLMs to generate hypothetical documents based on an initial query. Despite their simplicity, these methods have demonstrated substantial effectiveness across benchmarks for zero-shot retrieval and knowledge-intensive tasks Wang et al. (2024a) , including fact verification Yoon et al. (2024) . Data Leakage and LLM Memorization Previous research has investigated various forms of data leakage in LLMs Kandpal et al. (2023) ; Samuel et al. (2025) ; Deng et al. (2024) ; Xu et al. (2024a) . One study used perplexity to detect potential data leakage, uncovering substantial instances of training or even test set misuse Xu et al. (2024b) . Deng et al. (2023) further examined data contamination by predicting masked tokens in test sets and found that GPT 3.5 could reconstruct missing portions of MMLU Hendrycks et al. (2020) test instances with 57% accuracy. Other research has explored the boundaries of LLM knowledge Yin et al. (2023) ; Dong et al. (2024) ; Burns et al. (2022) ; Kadavath et al. (2022) ; Mallen et al. (2023) , including a refusal-aware instruction tuning method that trains LLMs to reject uncertain questions Zhang et al. (2024) —where an LLM is deemed uncertain if its generated response does not match the ground truth. Another study leveraged response consistency to estimate an LLM’s confidence in its knowledge Cheng et al. (2024) . In this work, we apply NLI MacCartney (2009) to LLM-generated documents, referencing gold evidence, to examine what an LLM knows. 3 Methodology 3.1 Task and Dataset Fact verification aims to predict the veracity label of a textual claim c . Depending on the dataset, the veracity label can fall into one of three or four categories 1 1 1 AVeriTeC provides four categories, whereas FEVER and SciFact use three, excluding conflicting evidence . In SciFact, CONTRADICT is treated equivalently to refuted . : supported , refuted , not enough evidence , or conflicting evidence . The fact-verification task consists of two sub-tasks: evidence retrieval and verdict prediction. In evidence retrieval, a retrieval method R(\cdot) identifies an evidence set \tilde{E}=\{\tilde{e_{1}},\cdots,\tilde{e_{k}}\} from a knowledge store K (e.g., Wikipedia), used to verify c . The performance of R is evaluated by comparing \tilde{E} with E=\{e_{1},\cdots,e_{l}\} , the gold evidence set. Verdict prediction then determines the veracity label of c based on \tilde{E} . We choose fact verification as the target task to test our hypothesis for two main reasons. First, in real-world fact-checking scenarios Miranda et al. (2019) ; Nakov et al. (2021) , retrieving evidence about niche or novel knowledge is crucial. If QE is effective only when the relevant knowledge has been seen during language model pretraining, its

Read the complete text