PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation
Albert Gong, Kamilė Stankevičiūtė, Chao Wan, Anmol Kabra, Raphael Thesmar, Johann Lee, Julius Klenke, Carla P. Gomes, Kilian Q. Weinberger · 2025
Read PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation by Albert Gong, Kamilė Stankevičiūtė, Chao Wan, Anmol Kabra, Raphael Thesmar, Johann Lee, Julius Klenke, Carla P. Gomes, Kilian Q. Weinberger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-20377.txt.
Description
Albert Gong and co-authors generate fresh test sets for AI reasoning on demand, so the tests cannot leak into training data, in work from 2025. Fixed benchmarks age badly: models memorize them and scores inflate.
PhantomWiki builds a brand-new, factually consistent mini-world of documents and questions for each evaluation, owing nothing to existing data. Turning knobs for question difficulty and corpus size separates reasoning skill from retrieval skill. Even frontier models found the generated sets surprisingly hard. The authors release the code and pitch the framework as a scalable way to measure reasoning, retrieval, and tool use without contamination, a standing problem for honest AI evaluation.
Catalog record
| CREATOR | Albert Gong, Kamilė Stankevičiūtė, Chao Wan, Anmol Kabra, Raphael Thesmar, Johann Lee, Julius Klenke, Carla P. Gomes, Kilian Q. Weinberger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-20377