Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP
Urja Khurana, Michiel van der Meer, Enrico Liscio, Antske Fokkens Pradeep K. Murukannaiah Department of Intelligent Systems, Delft University of Technology Leiden Institute of Advanced Computer Science (LIACS), Leiden University Computational Linguistics and Text Mining Lab, Vrije Universiteit Amsterdam {u.khurana, e.liscio, p.k.murukannaiah}@tudelft.nl,[email protected], [email protected] · 2026
Read Not All Subjectivity Is the Same! Defining Desiderata for the Evaluation of Subjectivity in NLP by Urja Khurana, Michiel van der Meer, Enrico Liscio, Antske Fokkens Pradeep K. Murukannaiah Department of Intelligent Systems, Delft University of Technology Leiden Institute of Advanced Computer Science (LIACS), Leiden University Computational Linguistics and Text Mining Lab, Vrije Universiteit Amsterdam {u.khurana, e.liscio, p.k.murukannaiah}@tudelft.nl,[email protected], [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28351.txt.
Description
Khurana and four co-authors argue that subjectivity in language data is not one thing, and judging it needs care. Models that honor many viewpoints can lift up minority voices, yet scoring often ignores that goal. The team lists seven wishes for good evaluation, drawn from how subjectivity shows up in data and models. A scan of 60 papers finds real gaps, like mixing up ambiguous and multi-voiced inputs. They urge checks that match what these models are for.
Catalog record
| CREATOR | Urja Khurana, Michiel van der Meer, Enrico Liscio, Antske Fokkens Pradeep K. Murukannaiah Department of Intelligent Systems, Delft University of Technology Leiden Institute of Advanced Computer Science (LIACS), Leiden University Computational Linguistics and Text Mining Lab, Vrije Universiteit Amsterdam {u.khurana, e.liscio, p.k.murukannaiah}@tudelft.nl,[email protected], [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40128 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.28351