Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead

Tom Sühr, Florian E. Dorner, Olawale Salaudeen, Augustin Kelava, Samira Samadi · 2025

Read Stop Evaluating AI with Human Tests, Develop Principled, AI-specific Tests instead by Tom Sühr, Florian E. Dorner, Olawale Salaudeen, Augustin Kelava, Samira Samadi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23009.txt.

Description

Tom Sühr and co-authors argue that testing AI systems with human exams is a basic error, in work from 2025. Language models now score high on tests of human smarts and personality. Those scores are often read as proof that models share human traits.

Human tests were built and tuned for human groups, with a theory of the trait behind them. Using them on machines without fresh validation confuses what is being measured. The authors also flag dirty test data, cultural skew, and wild swings from small prompt changes. They call for new tests built for AI itself, with clear theory and proof that scores mean what they claim.

Catalog record

CREATORTom Sühr, Florian E. Dorner, Olawale Salaudeen, Augustin Kelava, Samira Samadi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-23009

Browse the full library · About the Tower of Babel