Evaluating the Evaluator: Problems with SemEval-2020 Task 1 for Lexical Semantic Change Detection
Bach Phan-Tat, Kris Heylen, Dirk Geeraerts, Stefano De Pascale, Dirk Speelman · 2026
Read Evaluating the Evaluator: Problems with SemEval-2020 Task 1 for Lexical Semantic Change Detection by Bach Phan-Tat, Kris Heylen, Dirk Geeraerts, Stefano De Pascale, Dirk Speelman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13232.txt.
Description
Phan-Tat, Heylen, Geeraerts, De Pascale, and Speelmana re-examine SemEval-2020 Task 1, the leading shared benchmark for lexical semantic change detection, through operationalization, data quality, and benchmark design. They argue its discrete-sense framing misses gradual, constructional, collocational, and discourse-level change, while annotation choices and thresholds limit validity. Data problems include OCR noise, malformed characters, truncated sentences, inconsistent lemmatization, and tagging errors, and small curated target sets with narrow language coverage reduce realism.
Catalog record
| CREATOR | Bach Phan-Tat, Kris Heylen, Dirk Geeraerts, Stefano De Pascale, Dirk Speelman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29761 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.13232