Mapping Overlaps in Benchmarks through Perplexity in the Wild
Siyang Wu, Honglin Bao, Sida Li Ari Holtzman James A. Evans, SW, HB, and SL are co-leads on the project and contributed equally. Data Science Institute, University of Chicago{siyangwu, honglinbao, listar2000, jevans, aholtzman}@uchicago.edu · 2025
Read Mapping Overlaps in Benchmarks through Perplexity in the Wild by Siyang Wu, Honglin Bao, Sida Li Ari Holtzman James A. Evans, SW, HB, and SL are co-leads on the project and contributed equally. Data Science Institute, University of Chicago{siyangwu, honglinbao, listar2000, jevans, aholtzman}@uchicago.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-23488.txt.
Description
Benchmarks for language models overlap in ways that raw scores hide. Siyang Wu and 4 co-authors define benchmark signatures: sets of telling tokens drawn from real-world text. A model's perplexity on those tokens, which reflects its training exposure, predicts how it scores on a benchmark. The team built the signatures with stepwise selection and linear regression across many models and benchmarks.
In a study of thirty-two models and eighty-nine benchmarks, signatures showed deep overlap among knowledge and reasoning tests. Culture and humanity tests, by contrast, shared little with each other. Coding stood out as the most isolated skill. Only the knowledge signature matched real knowledge, hinting that model concepts may be organized unlike human ones.
Catalog record
| CREATOR | Siyang Wu, Honglin Bao, Sida Li Ari Holtzman James A. Evans, SW, HB, and SL are co-leads on the project and contributed equally. Data Science Institute, University of Chicago{siyangwu, honglinbao, listar2000, jevans, aholtzman}@uchicago.edu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-23488