On Benchmarking Human-like Intelligence in Machines
Lance Ying, Katherine M. Collins, Lionel Wong, Ilia Sucholutsky, Ryan Liu, Adrian Weller, Tianmin Shu, Thomas L. Griffiths, Joshua B. Tenenbaum · 2025
Read On Benchmarking Human-like Intelligence in Machines by Lance Ying, Katherine M. Collins, Lionel Wong, Ilia Sucholutsky, Ryan Liu, Adrian Weller, Tianmin Shu, Thomas L. Griffiths, Joshua B. Tenenbaum (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-20502.txt.
Description
Lance Ying and co-authors argue that today's AI tests fail to measure human-like thinking, in a 2025 paper. Models trained on human text are often cast as rough models of the human mind. The authors list the tests' flaws: labels never checked against people, no room for how human answers vary, and tasks too simple or too unlike real life.
To back the claim, they ran a study of people taking nine current AI benchmarks and found deep design problems. They close with five fixes for future tests, from human-checked labels to tasks drawn from real settings. The aim is a harder, fairer yardstick for claims about machine minds.
Catalog record
| CREATOR | Lance Ying, Katherine M. Collins, Lionel Wong, Ilia Sucholutsky, Ryan Liu, Adrian Weller, Tianmin Shu, Thomas L. Griffiths, Joshua B. Tenenbaum |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-20502