MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance
Joseph J. Peper, Wenzhao Qiu, Ali Payani, Lu Wang · 2025
Read MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance by Joseph J. Peper, Wenzhao Qiu, Ali Payani, Lu Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-14927.txt.
Introduction
Joseph Peper, Wenzhao Qiu, Ali Payani, and Lu Wang built a benchmark for reasoning across several documents in a 2025 study. Language models now read long inputs, but few tests rigorously check whether they can combine facts spread over multiple sources. Human annotation of such long tests is expensive, which slows benchmark creation.
MDBench is generated synthetically, with knowledge guidance steering the construction so questions stay grounded and answers checkable. Synthetic generation lets the authors control difficulty and coverage at scale. A trustworthy multi-document test matters because research assistants, legal tools, and analysts all depend on models that read beyond a single page.
Catalog record
| CREATOR | Joseph J. Peper, Wenzhao Qiu, Ali Payani, Lu Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-14927
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
MDBench: A Synthetic Multi-Document Reasoning Benchmark Generated with Knowledge Guidance Joseph J. Peper, Wenzhao Qiu, Ali Payani, Lu Wang Abstract Natural language processing evaluation has made significant progress, largely driven by the proliferation of powerful large language models (LLMs). New evaluation benchmarks are of increasing priority as the reasoning capabilities of LLMs are expanding at a rapid pace. In particular, while multi-document (MD) reasoning is an area of extreme relevance given LLM capabilities in handling longer-context inputs, few benchmarks exist to rigorously examine model behavior in this setting. Moreover, the multi-document setting is historically challenging for benchmark creation due to the expensive cost of annotating long inputs. In this work, we introduce MDBench , a new dataset for evaluating LLMs on the task of multi-document reasoning. Notably, MDBench is created through a novel synthetic generation process, allowing us to controllably and efficiently generate challenging document sets and the corresponding question-answer (QA) examples. Our novel technique operates on condensed structured seed knowledge, modifying it through LLM-assisted edits to induce MD-specific reasoning challenges. We then convert this structured knowledge into a natural text surface form, generating a document set and corresponding QA example. We analyze the behavior of frontier LLMs and prompting techniques, finding that MDBench poses significant challenges for all methods, even with relatively short document sets. We also see our knowledge-guided generation technique (1) allows us to readily perform targeted analysis of MD-specific reasoning capabilities and (2) can be adapted quickly to account for new challenges and future modeling improvements. Figure 1: MDBench generation pipeline overview. We 1) source structured knowledge, then use in-context multi-document reasoning demonstrations to 2) intentionally modify the existing knowledge with challenging dependencies. We then 3) map this seed knowledge into document form to produce the multi-document QA example. Finally, we 4) employ an added validity check to filter inconsistent examples that yield erratic outputs in an oracle setting where example construction knowledge is provided. 1 Introduction The rapid advancements in natural language processing (NLP) have been largely driven by the development and deployment of large language models (LLMs). These models have showcased remarkable improvements in various tasks, including understanding, generating, and reasoning over text. However, despite these advancements, evaluation frameworks for NLP systems have struggled to keep pace ( Chang et al. 2024 ) , notably for tasks involving reasoning over multiple documents ( Mavi et al. 2024 ) . Multi-document (MD) reasoning involves synthesizing and inferring information across multiple diverse texts ( Caciularu et al. 2021 ) , posing unique challenges not addressed by traditional single-document benchmarks. While LLMs are increasingly capable of handling longer-context multi-document inputs, there is a scarcity of benchmarks that rigorously examine the specific reasoning characteristics that are prominent in this setting. In addition, many existing benchmarks consist of static, hand-crafted datasets, which are labor-intensive to produce. These datasets are often susceptible to data contamination ( Xu et al. 2024 ) over time, e.g., LLMs are exposed to public benchmarks during training. This can compromise the integrity of the evaluation. In this work, we address these limitations with MDBench 1 1 1 Benchmark can be found at https://huggingface.co/launch/MDBench . Code and prompts are accessible at https://github.com/jpeper/MDBench . , a benchmark using a novel generation technique for multi-document reasoning evaluation. Our benchmark is generated through a synthetic process that leverages structured knowledge as seed information. This process uses a strong LLM (GPT-4o) to augment structured knowledge by injecting complexities that require advanced reasoning skills, then generates text documents from the augmented knowledge. Synthetic generation allows for efficient example creation and can be easily adapted to incorporate new reasoning skills and objectives. Our benchmark generation pipeline (overviewed in Figure 1 ) begins with a structured knowledge source serving as the seed information. Each knowledge entry (i.e., a row of a table) encapsulates distinct knowledge that forms the basis of a document in the generated set. We use tabular data as these are widely utilized and inherently structured, with row entries often sharing topical connections, making them an ideal foundation for evaluating cross-document reasoning. We follow a multi-step augmentation process to source knowledge, augment knowledge, and generate high-quality document sets with multi-document reasoning challenges: 1. Source Seed Knowledge (Fig. 1 , Step 1) We collect tabular data where each row contains information that will contribute to a generated document. 2. Augment Knowledge (Fig. 1 , Step 2) Using GPT-4o, we edit the structured knowledge to inject challenging reasoning dependencies and enrich the context for document creation. By treating rows as proxies for documents, we model cross-document dependencies through cross-row knowledge interactions. In this step, we also generate question-answer pairs that utilize the introduced reasoning dependencies. 3. Generate Natural Text (Fig. 1 , Step 3) We map the augmented knowledge into natural text by generating a corresponding multi-document set from the augmented table. This process allows us to systematically inject critical reasoning challenges while producing examples that are realistic and fluent. 4. Automated Quality Validation (Fig. 1 , Step 4) To ensure high quality and consistency, we apply targeted consistency prompts and a novel oracle-setting self-consistency check that effectively filters out low-quality examples, enabling rapid benchmark scaling. We produce 1,000 multi-document QA examples using this pipeline (300 human-validated, and 700 more automatically-validated for quality) and evaluate the performance of models from several prominent LLM families including GPT OpenAI et al. 2024 , Claude Anthropic 2024 , Gemini Team et al. 2024a , and LLaMA Dubey et al. 2024 . We find that: • MDBench poses a strong challenge, even for state-of-the-art methods, with the best ones achieving \sim 81% accuracy and \sim 62% exact-match performance on this MD reasoning task. • Frontier models such as GPT-4o, GPT-o1 and Gemini-2.5-Flash significantly outperform smaller LLMs across different prompting methods. This highlights the importance of model capacity and sophistication in handling complex multi-document reasoning tasks. • When comparing performance on document reasoning versus tabular reasoning (i.e., structured format pre-document generation), we find that strong models are mostly performant in both settings, but all struggle more in the document setting. This suggests that multi-document reasoning is influenced by both the fundamental reasoning complexity, and also from the nuances of the surface form . • Prompting techniques such as chain-of-thought ( Wei et al. 2022 ) can improve performance across strong models. However, they are insufficient to significantly enhance the performance of weaker models like LLaMA-3-7B and even GPT-3.5. This indicates that while prompting strategies can aid reasoning, underlying model capabilities remain a limiting factor for this task, which makes MDBench suitable for future, advanced model evaluation . 2 Related Work Evaluating the capabilities of LLMs is a critical aspect of NLP research. As LLMs continue to improve rapidly, existing evaluation frameworks often lag behind, particularly in assessing complex reasoning abilities such as multi-document (MD) reasoning. As LLMs rapidly increase in reasoning capacity, there is a pressing need to develop evaluation methods that can capture these higher-order reasoning skills. Multi-Document Reasoning MD reasoning involves synthesizing and inferring information across multiple texts. Existing work in this area includes datasets targeting specific phenomena such as temporal reasoning ( Xiong et al. 2024 ; Wan 2007 ) , summarization ( Xiao et al. 2021 ; Peper et al. 2023 ; Lior et