FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents

Nandan Thakur Jimmy Lin Sam Havens, Michael Carbin Omar Khattab Andrew Drozdov · 2025

Read FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents by Nandan Thakur Jimmy Lin Sam Havens, Michael Carbin Omar Khattab Andrew Drozdov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-13128.txt.

Introduction

Nandan Thakur and five co-authors build retrieval benchmarks in twenty twenty-five. Testing search systems needs hard, realistic questions over real documents. Their framework, FreshStack, builds such benchmarks automatically. It starts by collecting corpora from code and technical documentation.

Next it generates nuggets, the key facts inside community questions and answers. It then checks nugget-level support by retrieving documents with fused techniques. The team built five data sets on fast-growing, recent, niche topics. Those choices keep the tasks genuinely challenging. Existing retrieval models, used out of the box, fell well short of oracle performance on all five. Rerankers sometimes failed to help first-stage retrieval at all. Readers get headroom measured honestly for technical search.

Catalog record

CREATORNandan Thakur Jimmy Lin Sam Havens, Michael Carbin Omar Khattab Andrew Drozdov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-13128

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents Nandan Thakur Jimmy Lin Sam Havens, Michael Carbin Omar Khattab Andrew Drozdov Abstract We introduce FreshStack, a holistic framework for automatically building information retrieval (IR) evaluation benchmarks by incorporating challenging questions and answers. FreshStack conducts the following steps: (1) automatic corpus collection from code and technical documentation, (2) nugget generation from community-asked questions and answers, and (3) nugget-level support, retrieving documents using a fusion of retrieval techniques and hybrid architectures. We use FreshStack to build five datasets on fast-growing, recent, and niche topics to ensure the tasks are sufficiently challenging. On FreshStack, existing retrieval models, when applied out-of-the-box, significantly underperform oracle approaches on all five topics, denoting plenty of headroom to improve IR quality. In addition, we identify cases where rerankers do not improve first-stage retrieval accuracy (two out of five topics) and oracle context helps an LLM generator generate a high-quality RAG answer. We hope FreshStack will facilitate future work toward constructing realistic, scalable, and uncontaminated IR and RAG evaluation benchmarks. 1 Introduction Retrieval-augmented generation (RAG) is a popular technique to enhance traditional information retrieval (IR) capabilities with language model generation. RAG systems use large language models (LLMs) to generate long-form responses [ 20 , 34 , 23 , 5 ] , grounded in the information available from retrieved documents [ 31 , 34 , 18 , 41 ] . Despite its wide usage, evaluating RAG remains incredibly challenging. Existing IR and RAG benchmarks are not well-suited for evaluation, as these are outdated and highly limited. In particular, we observe three major issues in existing benchmarks: • Lack of realistic, open-ended questions : Existing datasets contain purely extractive short answers (e.g., Natural Questions [ 33 ] , TriviaQA [ 28 ] ) or crowd-sourced questions (e.g., HotPotQA [ 83 ] ). A limited number of datasets capture “natural” human-asked questions, i.e., MS MARCO [ 46 ] or Natural Questions [ 33 ] , but unfortunately, brief and straightforward questions are inserted into a search box, failing to represent the complex questions that real users might pose to RAG systems. • Artificially easy : RAG represents an approach rather than a problem . Real users require systems capable of grounded question answering, i.e., responding to specialized questions by referencing knowledge from a document corpus. Consequently, datasets constructed by design to be solvable via retrieval often fail to encode challenges faced in RAG applications. • Static and unspecialized : After sourcing questions and answers, a benchmark becomes at the risk of (1) contamination , if current LLMs are trained on the same set of documents or questions, (2) overfitting , when systems inevitably saturate by repeated internal or external leaderboarding (e.g., BEIR [ 68 ] ), and (3) staleness , when questions or answers are not refreshed and become outdated. Figure 1: A data instance from LangChain generated with FreshStack. The question and answer pair is sourced from Stack Overflow. The pair is provided to GPT-4o to generate nuggets, highlighting necessary facts in the answer. Next, code snippets and technical documents from multiple GitHub repositories (e.g., Jupyter Notebook) are chunked, processed, and pooled for each question. Finally, each pooled document chunk is judged with GPT-4o for binary relevance (either yes or no) at a nugget-level, i.e., whether the document factually supports the information present in each nugget. A realistic benchmark should measure model generalization on niche domains, and continue to update. Additionally, it must capture the complexity of human-generated queries—such as multi-hop reasoning [ 83 ] , code understanding [ 27 ] , or specialized terminology—rather than relying on artificially easy questions. This drives the robustness of systems in answering questions in evolving public libraries or private code bases [ 86 , 27 ] , a company’s internal forum [ 55 ] , or technical troubleshooting [ 64 , 53 , 8 ] . In our work, we introduce FreshStack , a holistic framework for constructing realistic datasets on niche and challenging domains, seeking to avoid contamination due to (perpetual) recency. Using FreshStack, we construct an evaluation benchmark on five niche topics sourced from community-asked questions and answers on Stack Overflow and a corpus containing code snippets and technical documents from public GitHub repositories. The framework contains three major steps: (1) Automatic corpus collection (Section 3.2 ) with technical documents chunked and sourced from several GitHub repositories. (2) Nugget generation (Section 3.3 ) with GPT-4o using community-asked questions and answers in Stack Overflow. (3) Nugget-level support (Section 3.5 ) with GPT-4o on document chunks, retrieved from a fusion of retrieval techniques and hybrid architectures. We investigate three research questions in our work to provide insights on FreshStack: (RQ1) How to construct challenging evaluation datasets from real user-asked questions? (RQ2) How do LLMs act as an assessor for nugget generation on community-asked questions & answers and nugget-level support with retrieved documents? (RQ3) How do state-of-the-art retrieval models, rerankers, and LLM generators perform on IR and RAG evaluation benchmarks generated with FreshStack? We calibrate the automatic stage in FreshStack with GPT-4o using a machine learning (ML) expert, assessing the quality of nugget generation and nugget-level support for one of the topics (LangChain). Our results show that GPT-4o-generated nuggets capture crucial information required to answer the question, and GPT-4o precisely labels support at a nugget level. For pooling, we compare oracle (having access to the answer) and inference (relying only on the question) settings, finding that question decomposition and nugget generation outperform other techniques, respectively. Beyond pool construction, we evaluate retrieval and rerankers in the document retrieval setting using only the Stack Overflow question. Retrieval models drastically underperform oracle systems on all five topics, showing a high headroom for improvement. In addition, ensemble fusion outperforms individual models, indicating that diversity in models enhances retrieval, and rerankers provide clear benefits in some but not all topics. Finally, we evaluate answer generation, where the oracle context assists the LLM generator to provide a high-quality RAG answer. FreshStack is a general framework and can be applied to any domain of a similar structure. Overall, we hope the framework serves as a testbed for future work to develop challenging benchmarks for evaluating RAG systems. 2 Related Work Retrieval-augmented generation. RAG has been widely used to avoid “hallucinations” [ 87 ] seen with LLMs when handling knowledge-intensive tasks [ 29 ] . RAG reduces factually incorrect generated content, leading to adoption in various commercial systems, e.g., Bing Search or Google AI Overviews. Existing IR and RAG benchmarks are stale, evaluating on academic question answering datasets [ 18 , 58 , 56 , 65 , 81 ] , or are not challenging, being constructed for RAG [ 82 , 7 , 37 , 47 , 65 , 32 ] . A limited number of datasets refresh over time to avoid LLM decontamination [ 30 , 74 , 62 , 79 ] , however, these contain easy and unrealistic questions. In contrast, FreshStack generates niche and challenging datasets, which can refresh over time and are not constructed specifically for RAG. Code-based benchmarks. Neural code generation [ 43 ] requires LLMs to generate code from scratch for generic programming questions. One popular benchmark is SWE-Bench [ 27 ] , which evaluates whether

Read the complete text