Grounding Generative Evaluations of Language Models in Unsupervised Document Corpora
Michael Majurski, Cynthia Matuszek · 2025
Read Grounding Generative Evaluations of Language Models in Unsupervised Document Corpora by Michael Majurski, Cynthia Matuszek (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-08905.txt.
Introduction
Michael Majurski and Cynthia Matuszek propose to build language model tests on their own from plain documents. Benchmarks gauge model quality. But human work to build them is outpaced by the size and scope of models under test. To build a human benchmark for each domain is not practical.
Their method builds fact-based fake tests grounded in sets of documents. It uses only grounding documents, such as a textbook, as input. The same language models make both multiple-choice and open-ended questions. The made benchmarks match human-made questions well. They show strong rank and accuracy links in the team's checks. The team used the method on three recent documents. Two came out after the knowledge cutoffs of the models. One model family showed strong scores, as a surprise. The path turns any set of documents into a test.
Catalog record
| CREATOR | Michael Majurski, Cynthia Matuszek |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 122 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-08905
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Grounding Generative Evaluations of Language Models in Unsupervised Document Corpora Michael Majurski, Cynthia Matuszek Abstract Language Models (LMs) continue to advance, improving response quality and coherence. Given Internet-scale training datasets, LMs have likely encountered much of what users may ask them to generate in some form during their training. A plethora of evaluation benchmarks have been constructed to assess model quality, response appropriateness, and reasoning capabilities. However, the human effort required for benchmark construction is rapidly being outpaced by the size and scope of the models under evaluation. Having humans build a benchmark for every possible domain of interest is impractical. Therefore, we propose a methodology for automating the construction of fact-based synthetic data model evaluations grounded in document populations. This work leverages the same LMs to evaluate domain-specific knowledge automatically, using only grounding documents (e.g., a textbook) as input. This generative benchmarking approach corresponds well with human curated questions producing an ensemble Spearman ranking correlation of 0.91 0.91 and a benchmark evaluation Pearson accuracy correlation of 0.74 0.74 (model specific 0.82 0.82 ). This novel approach supports generating both multiple choice and open-ended synthetic data questions to gain diagnostic insight of LM capability. We apply this methodology to evaluate model performance on three recent documents (two post LM knowledge cutoff), discovering a surprisingly strong performance from Gemma-3 models on open-ended questions. Code is available at https://github.com/mmajurski/grounded-synth-lm-benchmark 1 Introduction Figure 1 : Generative evaluation pipeline: (1) users curate grounding documents, (2) LMs generate domain-specific questions, (3) model responses are evaluated. Recent advances in language models (LMs) have demonstrated significant capability improvements in diverse natural language processing tasks. This expansion in capability has primarily been enabled by training on Internet-scale corpora. While LMs generate fluent, plausible text, the outputs are not always grounded in the facts of a situation. As AI tooling improves professional workflows, rigorous evaluation is imperative to understand the domain-specific model capability. Traditional multiple choice question (MCQ) benchmarks such as MMLU ( Hendrycks et al., 2020 ) offer valuable general insights into model performance, but are static, public, and limited in scope. The fact that these benchmarks are static and public—and therefore can appear in training data—is a significant weakness. Combined with the human effort required to construct new high quality tests ( Phan et al., 2025 ) , it is unlikely evaluation construction can keep pace with LM development. This scaling challenge is especially acute for domain-specific professional applications, where the general benchmarks provide limited insight. Unfortunately, creating expert-validated benchmarks for every domain is impractical. New automated methods must generate accurate, comprehensive, diagnostically relevant evaluations grounded in authoritative corpora with minimal human intervention beyond document selection. We propose an novel automated methodology for generating synthetic benchmarks grounded in authoritative documents. This approach reduces human effort and enables rapid adaptation to new domains by generating fact-based multiple choice or open-ended questions directly from appropriate grounding documents. This work is organized around three core questions: 1. How can LMs extract relevant information and generate high-quality questions grounded in domain documents? 2. Do these generative benchmarks replicate existing human-curated evaluations? 3. What limitations require mitigation for successful synthetic data evaluation of LM capability? 2 Related works Our approach is motivated by task asymmetry where given grounding documents, an LM should reliably generate diagnostically relevant questions, while it is substantially harder to zero-shot answer those same questions. This information asymmetry motivates using LMs to generate benchmarks grounded in curated documents rather than relying only on model weight knowledge. Language models are routinely adopted as judges: extracting answer choices ( Samuylova, 2024 ; Roucher, 2024 ) , grading open ended responses ( Ip, 2024 ; Oh et al., 2024 ) , and justifying correctness ( Cook et al., 2024 ) . The most similar contemporary convergent work is YourBench ( Shashidhar et al., 2025 ) synthetic replication of MMLU from grounding documents and AutoBencher ( Li et al., 2025b ) reinforcement learning optimization to discover gaps in LM knowledge. Leveraging grounding documents greatly improves synthetic data utility, as the methodology no longer relies on internalized LM knowledge. Modern long-context LMs can process entire documents. Benchmarks like SecQA ( Liu, 2023 ) and PubMedQA ( Jin et al., 2019 ) leverage domain-specific textbooks and web pages. Full-document grounding generally outperforms summary-based systems ( Bhat et al., 2023 ) . Continued LM improvement, prompting strategies ( Yadav et al., 2024 ) , and topic breadth ( Li et al., 2024 ) has further improved synthetic data. These methods also enhance downstream applications like retrieval-augmented generation ( Balaguer et al., 2024 ) . LMs can serve as acceptable feature extractors when guided by detailed prompts. Studies show strong correlation between LM-based and human judgments of text quality ( Chiang and Lee, 2023 ; Han et al., 2024 ) . This aligns with the LM-as-a-Judge paradigm used for open-ended benchmark evaluation ( Xu et al., 2023 ; Zheng et al., 2023 ) . Beyond open ended answer correctness, LMs can extract other textual features. However, limitations in LM ground truthing remain, especially in tasks like information retrieval where relevance judgments are more complex ( Soboroff, 2025 ; Szymanski et al., 2025 ) . LM evaluation benchmarks generally use either multiple choice (MC) or open-ended (OE) formats. Open-ended questions require synthesis ( Balepur et al., 2025 ) but depend on a judge LM for scoring ( Xu et al., 2023 ; Myrzakhan et al., 2024 ) , while MCQs are easier to grade but may inflate performance due to answer visibility ( Robinson et al., 2022 ; Balepur et al., 2024 ) . Frameworks like Inspect-AI ( UK AI Security Institute, 2024 ) support both types with prompt templates and answer shuffling. Both formats are sensitive to rewording, leading to inconsistent outputs ( Wang et al., 2024a ) , and newer frameworks aim to improve open-ended scoring precision ( Bernard et al., 2024 ; Kamalloo et al., 2023 ) . 3 Methods Constructing synthetic data benchmarks from relevant domain-specific documents requires chunking into contexts, topic extraction per context, question and answer generation, and finally an evaluation framework to determine model capability on the newly minted benchmark (as shown in Figure 2 ). All LM interactions use default generation parameters, except we set \text{temperature}=0.7 (o4-mini did not support setting temperature). Figure 2 : Generative benchmarking pipeline when used in production. During evaluation of the approach significantly more LM-Judge operations are included to evaluate question and answer quality. Document chunking into contexts While modern LMs have sufficient context windows to support whole document ingestion, chunking encourages the generated questions to be spread throughout the grounding document. An input document is first converted into markdown for interoperability with LMs. We use Docling ( Auer et al., 2024 ) for conversion from pdf to markdown, and pandoc ( MacFarlane, ) for conversion from html to markdown. Once in markdown, each document section is used as a context chunk for question generation. This chunking method is simple and relies on the document author crafting relevant sections. Enhancements to chunking and document data preparation will likely produce overall system improvements, but were not required to demonstrate baseline performance in this study. For instance, our approach only leverages the textual data; formatting and including