Evaluating the Evaluator: Problems with SemEval-2020 Task 1 for Lexical Semantic Change Detection

Bach Phan-Tat, Kris Heylen, Dirk Geeraerts, Stefano De Pascale, Dirk Speelman · 2026

Read Evaluating the Evaluator: Problems with SemEval-2020 Task 1 for Lexical Semantic Change Detection by Bach Phan-Tat, Kris Heylen, Dirk Geeraerts, Stefano De Pascale, Dirk Speelman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13232.txt.

Description

Phan-Tat, Heylen, Geeraerts, De Pascale, and Speelmana re-examine SemEval-2020 Task 1, the leading shared benchmark for lexical semantic change detection, through operationalization, data quality, and benchmark design. They argue its discrete-sense framing misses gradual, constructional, collocational, and discourse-level change, while annotation choices and thresholds limit validity. Data problems include OCR noise, malformed characters, truncated sentences, inconsistent lemmatization, and tagging errors, and small curated target sets with narrow language coverage reduce realism.

Catalog record

CREATORBach Phan-Tat, Kris Heylen, Dirk Geeraerts, Stefano De Pascale, Dirk Speelman
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29761
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.13232

Browse the full library · About the Tower of Babel