Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead
Tom Sühr, Florian E. Dorner, Olawale Salaudeen, Augustin Kelava, Samira Samadi · 2025
Read Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead by Tom Sühr, Florian E. Dorner, Olawale Salaudeen, Augustin Kelava, Samira Samadi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23009.txt.
Description
Tom Sühr and co-authors argue that testing AI systems with human exams is a basic error, in work from 2025. Language models now score high on tests of human smarts and personality. Those scores are often read as proof that models share human traits.
Human tests were built and tuned for human groups, with a theory of the trait behind them. Using them on machines without fresh validation confuses what is being measured. The authors also flag dirty test data, cultural skew, and wild swings from small prompt changes. They call for new tests built for AI itself, with clear theory and proof that scores mean what they claim.
Catalog record
| CREATOR | Tom Sühr, Florian E. Dorner, Olawale Salaudeen, Augustin Kelava, Samira Samadi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 59 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-23009