A large-scale evaluation of commonsense knowledge in humans and large language models

Tuan Dung Nguyen, Duncan J. Watts, Mark E. Whiting · 2025

Read A large-scale evaluation of commonsense knowledge in humans and large language models by Tuan Dung Nguyen, Duncan J. Watts, Mark E. Whiting (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10309.txt.

Introduction

Tuan Dung Nguyen, Duncan Watts and Mark Whiting run a large test of commonsense in people and in large language models, in their 2025 arXiv paper. Standard tests assume human common sense is uniform, scoring models against fixed ground-truth labels. Yet people vary a lot in what they find obvious, so those labels may reflect only their designers.

The authors instead measure how well a model's judgments match those of a whole human population. Treated as single respondents, most models scored below the human median. Used as simulators of a population, models matched real agreement patterns only modestly. Smaller open models were surprisingly competitive with larger frontier ones.

Catalog record

CREATORTuan Dung Nguyen, Duncan J. Watts, Mark E. Whiting
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE154 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-10309

Browse the full library · About the Tower of Babel