OKBench: Democratizing LLM Evaluation with Fully Automated, On-Demand, Open Knowledge Benchmarking
Yanhong LiUniversity of Chicago [email protected], Tianyang XuTTI-Chicago [email protected], Kenan TangUC Santa Barbara [email protected] Karen LivescuTTI-Chicago [email protected] David McAllesterTTI-Chicago [email protected] Jiawei ZhouStony Brook University [email protected] · 2025
Read OKBench: Democratizing LLM Evaluation with Fully Automated, On-Demand, Open Knowledge Benchmarking by Yanhong LiUniversity of Chicago [email protected], Tianyang XuTTI-Chicago [email protected], Kenan TangUC Santa Barbara [email protected] Karen LivescuTTI-Chicago [email protected] David McAllesterTTI-Chicago [email protected] Jiawei ZhouStony Brook University [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-08598.txt.
Introduction
Yanhong Li and five co-authors build a benchmark machine for knowledge questions that never sits still. Static test sets drawn from textbooks go stale as the world changes, so their framework, OKBench, generates fresh benchmarks on demand, focusing on news where facts update daily.
Software agents handle the whole pipeline: sourcing material, writing questions, validating them, and distributing the finished test. Because the questions are new, they barely overlap with what models memorized in training, which also makes the tests fair for retrieval-augmented systems. Trying many open and commercial models, the team finds retrieval shrinks the gap between small and large models. Behavior on fresh facts differs sharply from behavior on memorized ones.
Catalog record
| CREATOR | Yanhong LiUniversity of Chicago [email protected], Tianyang XuTTI-Chicago [email protected], Kenan TangUC Santa Barbara [email protected] Karen LivescuTTI-Chicago [email protected] David McAllesterTTI-Chicago [email protected] Jiawei ZhouStony Brook University [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-08598
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OKBench: Democratizing LLM Evaluation with Fully Automated, On-Demand, Open Knowledge Benchmarking Yanhong LiUniversity of Chicago [email protected], Tianyang XuTTI-Chicago [email protected], Kenan TangUC Santa Barbara [email protected] Karen LivescuTTI-Chicago [email protected] David McAllesterTTI-Chicago [email protected] Jiawei ZhouStony Brook University [email protected] Abstract Knowledge-intensive question answering is central to large language models (LLMs) and is typically assessed using static benchmarks derived from sources like Wikipedia and textbooks. However, these benchmarks fail to capture evolving knowledge in a dynamic world, and centralized curation struggles to keep pace with rapid LLM advancements. To address these drawbacks, we propose Open Knowledge Bench ( OKBench ), a fully automated framework for generating high-quality, dynamic knowledge benchmarks on demand. Focusing on the news domain where knowledge updates daily, OKBench is an agentic framework that automates the sourcing, creation, validation, and distribution of benchmarks. Our approach democratizes benchmark creation and facilitates thorough evaluation of retrieval-augmented methods by reducing overlap with pretraining data. We evaluate our framework on a wide range open-source and proprietary LLMs of various sizes and configurations, both with and without retrieval over freshly generated knowledge. Our results reveal distinct model behaviors when confronted with new information and highlight how retrieval narrows the performance gap between small and large models. These findings underscore the importance of evaluating LLMs on evolving knowledge benchmarks. 1 Introduction One of the most common uses of large language models (LLMs) is for answering knowledge-intensive questions. However, this task is challenging as factual knowledge in the real world evolves rapidly. Well-trained models can quickly become outdated ( Li et al. 2024 ) , raising the need for continual model updates ( Liska et al. 2022 ) or improved retrieval-augmented generation (RAG) techniques ( Lewis et al. 2020 ) . At the same time, the lack of transparency in training data makes it difficult to assess how recent a model’s knowledge truly is ( Cheng et al. 2024 ) . Existing benchmarks also struggle to keep pace: once released, their contents may be absorbed into future training data, weakening their utility and leading to benchmark contamination. This phenomenon complicates the evaluation of retrieval-based methods, as models may have already memorized the relevant facts during training. In this paper, we propose that the solution to these challenges is fast, automated, decentralized curation of dynamic knowledge benchmarks that can track LLM development in real time and offer a clean testbed for evaluating retrieval augmented methods. Despite the rapid advancement of LLMs and the growing need for accurate knowledge assessment, most popular benchmarks remain static after creation. Widely used datasets such as Natural Questions ( Kwiatkowski et al. 2019 ) , TriviaQA ( Joshi et al. 2017 ) , and HotpotQA ( Yang et al. 2018 ) are primarily drawn from Wikipedia or curated text snapshots over a fixed time period. While instrumental in advancing open-domain question answering (QA) research, these benchmarks quickly become outdated and are often included in model pretraining corpora, leading to data contamination and inflated performance estimation ( Li et al. 2024 ) . More recent efforts such as StreamingQA ( Liska et al. 2022 ) , RealTimeQA ( Kasai et al. 2023 ) and FreshQA ( Vu et al. 2024 ) have begun including fresh facts. However, these dynamic benchmarks still rely on partial human curation and infrequent updates, or focus on a different task like forecasting. As a result, they don’t enable continuous evaluation of LLMs’ capabilities. Finally, these previous efforts are centralized , making it difficult and expensive to reproduce them on demand. Table 1: Comparison of our benchmark with some previous knowledge QA & dynamic benchmarks in terms of objective, automation, update frequency, and scale. Benchmark Objective Automation Update Freq. Scale StreamingQA Factual QA Partial Static 36,800 QA pairs RealTime QA Factual QA Partial Weekly \sim 30 QA pairs FreshQA Factual QA & Debunking Low Weekly 600 QA pairs, only update answers LiveBench Reasoning Partial Monthly 40-100 questions per task Daily Oracle Forecasting Full Daily \sim 17.2 QA pairs FutureX Future Event Prediction Full Daily & Weekly 500 events Ours Factual QA Full Any time \sim 2000 QA pairs We propose an approach that addresses these challenges and democratizes dynamic knowledge benchmarking, by making it easy and practical for anyone to generate a new reproducible benchmark anytime. Specifically, we introduce OKBench ( O pen K nowledge Bench ), a fully automated framework for generating knowledge benchmarks for fair LLM evaluation. Focusing on the news domain where new knowledge emerges daily, our system automates the entire pipeline from information extraction to benchmark construction, producing multiple - choice QA items (with optional open - ended variants). OKBench is an agentic framework built on state-of-the-art LLMs, in which specialized agents for QA generation and validation collaborate to promote quality and consistency. To enable benchmark generation at any time, we introduce a distribution and version control protocol that assigns each benchmark a unique signature, assuring consistent tracking and fair comparison across models and evaluations. The framework is fully open-source , empowering any user to generate up-to-date benchmarks at any time . This enables diverse use cases such as monitoring LLM knowledge freshness or evaluating retrieval-augmented models on clean, non-memorized data. To assess the quality of the automatically generated benchmarks, we conduct manual validation of one of our question sets and find it to be of high quality. To demonstrate the utility of our framework and assess current model capabilities, we evaluate a range of open-source and proprietary LLMs across multiple model sizes, with and without retrieval augmentation, using several retrieval strategies. Our results show a predictably large drop in performance when models are tested on new knowledge. Interestingly, when retrieval is introduced, the performance gap between smaller and larger models narrows significantly on knowledge not seen during training. 2 Related Work Evaluating Retrieval-Augmented Generation (RAG) Retrieval-augmented generation (RAG) is a key strategy to equip large language models (LLMs) with up-to-date information by retrieving relevant external documents at inference time. However, existing RAG evaluations are often undermined by data contamination , where evaluation examples overlap with the model’s pretraining corpus. This allows models to answer without retrieval, simply relying on memorized content ( Li et al. 2024 ) . Prominent QA datasets such as Natural Questions ( Kwiatkowski et al. 2019 ) , TriviaQA ( Joshi et al. 2017 ) , and HotpotQA ( Yang et al. 2018 ) are sourced from Wikipedia or the open web, making it likely that models already “know” the answers. This undermines robust assessment of retrieval: models can appear strong simply by regurgitating seen content, and including training examples in prompts can further inflate performance ( Wang et al. 2022 ) . As a result, current benchmarks struggle to test whether models can truly retrieve and reason over novel information. Figure 1: Automated dynamic knowledge benchmark construction pipeline for OKBench . Dynamic Knowledge Question Answering To address the limitations of static benchmarks, recent work has introduced dynamic QA benchmarks that reflect the evolving state of world knowledge. 1 1 1 For detailed descriptions of each benchmark, see Appendix A . StreamingQA ( Liska et al. 2022 ) organizes questions chronologically over years of news data, but it does not support continual updates. RealTime QA ( Kasai et al. 2023 ) delivers weekly