When the LM misunderstood the human chuckled : Analyzing garden path effects in humans and language models

Samuel Joseph Amouyal, Aya Meltzer-Asscher, Jonathan Berant λ \lambda Blavatnik School of Computer Science, Tel Aviv University, Israel † \dagger Department of Linguistics, Tel Aviv University, Israel ⋆ \star Sagol School of Neuroscience, Tel Aviv University, Israel {samuel.amouyal, joberant}.cs.tau · 2025

Read When the LM misunderstood the human chuckled : Analyzing garden path effects in humans and language models by Samuel Joseph Amouyal, Aya Meltzer-Asscher, Jonathan Berant λ \lambda Blavatnik School of Computer Science, Tel Aviv University, Israel † \dagger Department of Linguistics, Tel Aviv University, Israel ⋆ \star Sagol School of Neuroscience, Tel Aviv University, Israel {samuel.amouyal, joberant}.cs.tau (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09307.txt.

Description

Samuel Amouyal, Aya Meltzer-Asscher, and Jonathan Berant compare readers in twenty twenty-five. Garden-path sentences mislead readers into a wrong parse, then force a re-read. Humans find them notoriously hard. The authors ask whether large language models stumble in the same places, and for the same reasons.

They turn psycholinguistic hypotheses into tests. Human participants and a large suite of models answered comprehension questions on garden-path items. Both groups struggled with specific syntactic complexities. Some models correlated strongly with human comprehension patterns. Paraphrase and image generation tasks gave matching results. Readers see where machine parsing mirrors human parsing, and where the resemblance breaks. The study treats models as subjects in a classic language experiment.

Catalog record

CREATORSamuel Joseph Amouyal, Aya Meltzer-Asscher, Jonathan Berant λ \lambda Blavatnik School of Computer Science, Tel Aviv University, Israel † \dagger Department of Linguistics, Tel Aviv University, Israel ⋆ \star Sagol School of Neuroscience, Tel Aviv University, Israel {samuel.amouyal, joberant}.cs.tau
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-09307

Browse the full library · About the Tower of Babel