Is This Collection Worth My LLM’s Time? Automatically Measuring Information Potential in Text Corpora
Tristan KarchEPFL / DHLab [email protected], Luca EngelEPFL / DHLabPhilippe SchwallerEPFL / ILIACFrédéric KaplanEPFL / DHLab · 2025
Read Is This Collection Worth My LLM’s Time? Automatically Measuring Information Potential in Text Corpora by Tristan KarchEPFL / DHLab [email protected], Luca EngelEPFL / DHLabPhilippe SchwallerEPFL / ILIACFrédéric KaplanEPFL / DHLab (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13691.txt.
Introduction
Tristan Karch and co-authors ask a practical 2025 question: how can we tell if a text collection is worth feeding to a large language model? Digitizing and preparing collections costs a lot, yet models now differ less in skill, so new information matters more. Their method needs no model training.
The pipeline makes multiple choice questions from a collection's texts. It then tests a model with and without access to the source, and the gap in scores stands in for the collection's value. The team checked the method on five chosen data sets. Funders and curators get a cheaper way to judge collections before investing.
Catalog record
| CREATOR | Tristan KarchEPFL / DHLab [email protected], Luca EngelEPFL / DHLabPhilippe SchwallerEPFL / ILIACFrédéric KaplanEPFL / DHLab |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-13691
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Is This Collection Worth My LLM’s Time? Automatically Measuring Information Potential in Text Corpora Tristan KarchEPFL / DHLab [email protected], Luca EngelEPFL / DHLabPhilippe SchwallerEPFL / ILIACFrédéric KaplanEPFL / DHLab Abstract As large language models (LLMs) converge towards similar capabilities, the key to advancing their performance lies in identifying and incorporating valuable new information sources. However, evaluating which text collections are worth the substantial investment required for digitization, preprocessing, and integration into LLM systems remains a significant challenge. We present a novel approach to this challenge: an automated pipeline that evaluates the potential information gain from text collections without requiring model training or fine-tuning. Our method generates multiple choice questions (MCQs) from texts and measures an LLM’s performance both with and without access to the source material. The performance gap between these conditions serves as a proxy for the collection’s information potential. We validate our approach using five strategically selected datasets: EPFL PhD manuscripts, a private collection of Venetian historical records, two sets of Wikipedia articles on related topics, and a synthetic baseline dataset. Our results demonstrate that this method effectively identifies collections containing valuable novel information, providing a practical tool for prioritizing data acquisition and integration efforts. 1 Introduction Recent advances in large language models (LLMs) have revealed a striking phenomenon: as these models scale, they tend to develop remarkably similar internal representations and capabilities regardless of their architecture Huh et al. 2024 . This convergence, coupled with established scaling laws Kaplan et al. 2020 and the growing recognition that AI development should shift from model-centric to data-centric approaches Zha et al. 2025 , suggests that the key to advancing LLM capabilities lies in identifying and incorporating novel, high-quality information sources rather than architectural innovations. However, identifying valuable text collections for model enhancement presents significant challenges: digitization and preprocessing are costly, and training or fine-tuning models on new data requires substantial computational resources. This creates a critical need: how can we efficiently evaluate whether a text collection contains information that would meaningfully expand an LLM’s knowledge? Current approaches typically require actually training or fine-tuning models on new data to assess its value - an expensive and time-consuming process. While retrieval-augmented generation (RAG) Lewis et al. 2020 offers a promising approach for helping models access long-tail knowledge without full retraining Kandpal et al. 2023 , this still requires careful curation of knowledge bases and comes with its own computational costs. Moreover, the challenge of identifying valuable text collections remains: digitization and preprocessing are costly, and organizations need ways to evaluate the information potential of document collections before investing in their integration. We present a novel approach to this challenge: an automated pipeline that evaluates the potential information gain from text collections without requiring model training or fine-tuning. Our method generates and leverages multiple choice questions (MCQs) to systematically probe whether the knowledge contained in a text collection is already accessible to an LLM or represents genuinely novel information. To rigorously validate our approach, we evaluate it across three strategically selected datasets: (1) EPFL PhD manuscripts, containing specialized academic knowledge likely novel to LLMs, (2) Wikipedia articles that were presumably part of the LLMs’ training data, and (3) a synthetic baseline dataset composed of model-generated texts. Our key contributions are: 1. A novel end-to-end pipeline that uses MCQs to efficiently evaluate the potential information gain from text collections without requiring model training or fine-tuning 2. An automated filtering methodology for ensuring MCQ quality through complementary similarity metrics that address two key challenges: verifying question relevance and grounding in source material, while ensuring distractors are plausible yet unambiguously incorrect 3. A systematic analysis of knowledge gaps across diverse text collections that identifies areas of model uncertainty, quantifies the potential value of different information sources, and guides strategic decisions about data collection and integration Importantly, our approach is dataset-agnostic and can be applied to any text corpus. This makes it applicable to different domains and research questions. This pipeline could be leveraged to select textual data for model enhancements through fine-tuning or retrieval-augmented generation systems. 2 Related Work Information Gain. The challenge of efficiently selecting new information sources for LLMs can be viewed through the lens of optimal experiment design, a field pioneered by Fedorov et al. 1972 . This framework emphasizes maximizing information gain while being strategic about resource allocation – a particularly relevant consideration given the costs associated with integrating new data into LLM systems. Information gain itself has been conceptualized across various fields: in information theory, it relates to reductions in algorithmic information content Cover et al. 1989 ; in machine learning, it quantifies a feature’s contribution to model performance Odhiambo Omuya et al. 2021 ; and in cognitive science, it represents uncertainty reduction in our experience of the world Damiano et al. 2021 . While these theoretical frameworks provide valuable insights, they have not been previously applied to the specific challenge of evaluating the potential value of text collections for enhancing LLM knowledge. Our work bridges this gap by proposing a practical, MCQ-based approach that quantifies information gain by measuring an LLM’s ability to answer questions about a text collection with and without access to the source material. Knowledge Detection in LLMs. Prior research has developed several methods to analyze how LLMs process and retain textual information. Work on memorization Hartmann et al. 2023 ; Shi et al. 2024b and data contamination Yax et al. 2024 ; Golchin and Surdeanu 2024 focuses on identifying verbatim recall of training data, while hallucination detection Farquhar et al. 2024 aims to identify when models generate false information. Research on novelty detection has primarily focused on linguistic and semantic novelty McCoy et al. 2023 ; Lin et al. 2024 , with less attention paid to factual novelty. While these approaches provide valuable insights into model behavior, they are retrospective – analyzing what models have already learned or memorized. In contrast, our work takes a prospective approach, developing metrics to evaluate the potential value of new information sources before investing in their integration into LLM systems. 3 Methods Multiple Choice Questions (MCQs) are a well-established tool for knowledge assessment, supported by research in cognitive science and educational psychology Thomas M. Haladyna and Rodriguez 2002 . Their four-option format, consisting of one correct answer and three distractors, offers an optimal balance between assessment reliability and cognitive load Vyas and Supe 2007 . MCQs are particularly valuable for automated evaluation as they provide objective correctness measures while efficiently testing understanding across diverse topics Oc and Hassen 2024 . When working with Large Language Models (LLMs), MCQs offer an additional advantage: they constrain the output space to a finite set of options, eliminating the ambiguity and variability inherent in open-ended responses and enabling precise evaluation of model knowledge. Generating high-quality MCQs presents unique challenges, particularly due to the absence of ground truth signals typically available in human-curated educational assessments. Without explicit supervision on what constitutes a good question or appropriate distractors, we must derive reliable metrics to filter and validate the generated MCQs. This automated filtering setting raises two critical challenges: ensuring questions are both relevant and non-trivial, and guaranteeing that distractors are plausible yet unambiguously incorrect. The latter is particularly crucial when evaluating LLMs – if distractors