WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorization

Ine Gevers, Victor De Marez, Luna De Bruyne, Walter Daelemans · 2025

Read WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorization by Ine Gevers, Victor De Marez, Luna De Bruyne, Walter Daelemans (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23779.txt.

Description

Ine Gevers and co-authors paraphrase a famous commonsense benchmark and watch language model scores fall sharply, in work from twenty twenty-five. WinoGrande sentences test pronoun sense through twin-sentence choices. Their WinoWhat corpus rewrites every validation item in new words. Five commonsense categories tag what kind of knowledge each item needs. Models of many sizes face both versions of the test.

Every model performs notably worse on the paraphrased corpus. That gap suggests benchmark scores overstate true reasoning skill. Training-data matches are checked to rule out simple memorisation. Memorisation, it turns out, explains little of the score gap. Surface wording, not knowledge, props up headline results. Paraphrase twins make a sterner test for machine commonsense.

Catalog record

CREATORIne Gevers, Victor De Marez, Luna De Bruyne, Walter Daelemans
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-23779

Browse the full library · About the Tower of Babel