LLMpedia: A Transparent Framework to Materialize an LLM’s Encyclopedic Knowledge at Scale
Muhammed Saeed, Simon Razniewski · 2026
Read LLMpedia: A Transparent Framework to Materialize an LLM’s Encyclopedic Knowledge at Scale by Muhammed Saeed, Simon Razniewski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-24080.txt.
Introduction
Saeed and Razniewski present LLMpedia, a framework that writes encyclopedia articles purely from a language model's memory. They generate about 1.3 million articles across three model families. Every claim is then checked against Wikipedia and curated web evidence. For gpt-5-mini, only 68.4 percent of claims on Wikipedia-covered topics verify as true, far below MMLU-style scores. Most of the gap comes from claims that cannot be verified, not from flat errors. All prompts, articles, and verdicts are released for inspection.
Catalog record
| CREATOR | Muhammed Saeed, Simon Razniewski |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79781 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.24080
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLMpedia: A Transparent Framework to Materialize an LLM’s Encyclopedic Knowledge at Scale Muhammed Saeed, Simon Razniewski Abstract Benchmarks such as MMLU suggest flagship language models approach factuality saturation, with scores above 90%. We show this picture is incomplete. LLMpedia generates encyclopedic articles entirely from parametric memory, producing ∼ {\sim} 1M articles across three model families without retrieval. For gpt-5-mini, the verifiable true rate on Wikipedia-covered subjects is only 74.7% – more than 15 percentage points below the benchmark-based picture, consistent with the availability bias of fixed-question evaluation. Beyond Wikipedia, frontier subjects verifiable only through curated web evidence fall further to 63.2% true rate. Wikipedia covers just 61% of surfaced subjects, and three model families overlap by only 7.3% in subject choice. In a capture-trap benchmark inspired by prior analysis of Grokipedia, LLMpedia achieves substantially higher factuality at roughly half the textual similarity to Wikipedia. Unlike Grokipedia, every prompt, artifact, and evaluation verdict is publicly released, making LLMpedia the first fully open parametric encyclopedia – bridging factuality evaluation and knowledge materialization. All data, code, and a browsable interface are at https://llmpedia.net . 1 Introduction Large language models are now used as everyday knowledge interfaces by hundreds of millions of people 20 . Yet what these models actually know – and how reliably they express it in long-form text – remains poorly understood. Two research threads address this from opposite directions: factuality evaluation probes models on experimenter-chosen questions; knowledge materialization lets models produce their own knowledge at scale. Neither alone suffices: benchmarks miss what they never ask, and materializations lack systematic evaluation. LLMpedia bridges the two. Most current evaluation relies on curated benchmarks such as MMLU 8 , TruthfulQA 14 , and HLE 23 . These are indispensable, but they share a structural limitation: they test only knowledge the evaluator thought to ask about. This creates a form of availability bias 30 : a model can look highly factual on a fixed set of questions while containing broad regions of weak or unstable knowledge the benchmark never reaches. The limitation becomes sharper in real use, where people receive fluent, multi-paragraph explanations rather than short benchmark answers 24 . Figure 1: LLMpedia generates entirely from parametric memory with full auditability. Grokipedia’s opaque pipeline shows evidence of retrieval-shaped generation 36 . A natural alternative is to materialize model knowledge at scale. 2 and GPTKB 10 showed that recursive elicitation can surface vast amounts of model knowledge, including large regions absent from Wikipedia 9 ; 4 . But GPTKB materializes triples – a format unlike how users actually interact with models 18 – and its pipeline suffers from persistent entity disambiguation failures 4 ; 9 . We instead materialize full encyclopedic articles in the free-text modality users see, making it possible to evaluate subject choice, discourse structure, entity salience, and – crucially – how much apparent knowledge survives outside the narrow corridor of benchmark questions. This gap is large, for gpt-5-mini, the verifiable true rate on Wikipedia-covered subjects is only 74.7% – more than 15 percentage points below the 90%+ picture suggested by benchmark-style evaluation. That gap is the main empirical signal of this paper: fixed-question benchmarks overstate open-ended factual reliability in part because they are shaped by availability bias in what gets asked. Among surfaced subjects absent from Wikipedia but recoverable through curated web evidence, the true rate falls further to 63.2%. At the same time, Wikipedia covers only 61% of surfaced subjects, and three model families overlap on just 7.3%, indicating that materialized parametric knowledge is far more model-specific than leaderboard-style evaluation implies. This question matters beyond benchmarking. Wikipedia’s coverage is uneven, with a well-documented Western-centric bias 33 , gender skew 32 , and non-Anglophone gaps. If models encode reliable knowledge beyond Wikipedia, materializing it could help complement those gaps – but only if the generation process is auditable. The most visible LLM-generated encyclopedia, Grokipedia 35 , is methodologically opaque. Prior work reports high textual similarity to Wikipedia 36 , and we document additional evidence consistent with retrieval-shaped generation (Appendix K ). This creates a capture trap : the encyclopedia may inherit Wikipedia’s phrasing while obscuring both genuine model knowledge and its error modes. We present LLMpedia , a fully auditable, open-source framework for parametric encyclopedia generation. Starting from a seed, the system expands breadth-first through article generation and wikilink extraction, then filters candidates through a three-stage sanitization pipeline before re-enqueuing them. Every prompt, intermediate artifact, and evaluation verdict is publicly released. We operate in two modes: general-domain materialization , where BFS expands freely from a seed entity, and topic-focused generation , where expansion is constrained to entities associated with one of three root topics. Both regimes use prompt templates documented in App. J . Across gpt-5-mini 19 , Llama-3.3-70B-Instruct 6 , and DeepSeek-V3 15 , we produce approximately one million articles. Motivated by claims of ideological bias in Grokipedia, we additionally inject controlled editorial personas (scientific-neutral, left-leaning, conservative) across all pipeline stages in topic-focused runs, treating framing as an explicit, auditable variable rather than a hidden confounder. System Param. Fact. Full Art. Serendip. Scale Benchmarks MMLU 8 ✓ ✗ ✗ 10^{4} HLE 22 ✓ ✗ ✗ 10^{3} Knowledge materialization GPTKB 10 ✓ ✗ ✓ 10^{6} Cosmopedia 1 ✓ ✗ (✓) 10^{7} Encyclopedia generation WikiSum 16 ✗ ✓ ✗ 10^{4} STORM 27 ✗ ✓ ✗ 10^{2} Grokipedia 35 1 1 1 as of now Grokipeida v.1 about 1.06M ✗ (?) ✓ ? 10^{6} LLMpedia (ours) ✓ ✓ ✓ 10^{6} Table 1: Comparison across paradigms: benchmark-based probing, large-scale knowledge materialization, and encyclopedia generation. Param.=parametric-only generation (no retrieval at inference time), Fact. Full Art.=generates factual full text, Serendip.=enables not preconceived knowledge discovery. Our investigation centers on three questions: RQ1 : What do topic-focused parametric encyclopedias reveal about cross-model factuality, coverage, and the effect of editorial persona? RQ2 : At 10^{5} scale, how do three model families differ in knowledge breadth, overlap, and reliability? Does factuality degrade with BFS depth? RQ3 : Does parametric generation escape the Wikipedia capture trap – producing factually competitive but structurally independent content? Our core contributions are: 1. A scalable, fully open-source pipeline for parametric encyclopedia construction – bridging factuality evaluation and knowledge materialization into a single framework; 2. Evidence that open-ended parametric generation is substantially weaker than benchmark-style factuality suggests: for gpt-5-mini, true rate falls to 74.7% on Wikipedia-covered subjects, more than 15 points below the benchmark-based picture; 3. Evidence that the beyond-Wikipedia frontier is both real and difficult: subjects verifiable only through curated web evidence reach just 63.2% true rate; 4. A capture-trap benchmark, inspired by 36 , in which we collect the most-edited English Wikipedia articles and show that LLMpedia achieves higher factuality than Grokipedia at roughly half the textual similarity to Wikipedia. 2 Related Work Knowledge probing and its limits. Probing frameworks such as LAMA 21 test recall via cloze prompts; multitask suites like MMLU 8 span knowledge-heavy domains; TruthfulQA 14 targets misconceptions; HLE 22 pushes frontier difficulty. These measure only what the experimenter thought to ask. LLMpedia complements benchmarks by materializing knowledge at scale and decomposing outputs into true, false, and unverifiable. From triples to articles. 2 introduced recursive elicitation via knowledge-graph expansion; 10 scaled this into GPTKB (100M+ triples), revealing that many materialized entities are