PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation

Albert Gong, Kamilė Stankevičiūtė, Chao Wan, Anmol Kabra, Raphael Thesmar, Johann Lee, Julius Klenke, Carla P. Gomes, Kilian Q. Weinberger · 2025

Read PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation by Albert Gong, Kamilė Stankevičiūtė, Chao Wan, Anmol Kabra, Raphael Thesmar, Johann Lee, Julius Klenke, Carla P. Gomes, Kilian Q. Weinberger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-20377.txt.

Description

Albert Gong and co-authors generate fresh test sets for AI reasoning on demand, so the tests cannot leak into training data, in work from 2025. Fixed benchmarks age badly: models memorize them and scores inflate.

PhantomWiki builds a brand-new, factually consistent mini-world of documents and questions for each evaluation, owing nothing to existing data. Turning knobs for question difficulty and corpus size separates reasoning skill from retrieval skill. Even frontier models found the generated sets surprisingly hard. The authors release the code and pitch the framework as a scalable way to measure reasoning, retrieval, and tool use without contamination, a standing problem for honest AI evaluation.

Catalog record

CREATORAlbert Gong, Kamilė Stankevičiūtė, Chao Wan, Anmol Kabra, Raphael Thesmar, Johann Lee, Julius Klenke, Carla P. Gomes, Kilian Q. Weinberger
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-20377

Browse the full library · About the Tower of Babel