Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments
Amir Homayounirad, Enrico Liscio, Tong Wang, Luciano C. Siebert · 2025
Read Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments by Amir Homayounirad, Enrico Liscio, Tong Wang, Luciano C. Siebert (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06704.txt.
Introduction
Claims that rest on human values will be read in unlike ways by unlike people. Amir Homayounirad and four co-authors study how to foresee that split in a 2025 language technology paper. Labeling work usually merges many annotators into one ground truth. That merger hides personal judgment, which matters when the labels concern values.
Two plans are weighed. One infers personal judgment sideways, from value guesses; the other spots it head-on. Spotting it head-on flagged value-laden claims far better. Adding a contrastive loss, which pulls similar cases together and pushes different ones apart, to the standard sorting loss did not raise scores, though it cut reliance on per-label judgment data. The methods help annotation teams spot claims that people may read in unlike ways, backing a more careful labeling process that keeps the split visible instead of averaging it away.
Catalog record
| CREATOR | Amir Homayounirad, Enrico Liscio, Tong Wang, Luciano C. Siebert |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-06704
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Will Annotators Disagree? Identifying Subjectivity in Value-Laden Arguments Amir Homayounirad, Enrico Liscio, Tong Wang, Luciano C. Siebert Abstract Aggregating multiple annotations into a single ground truth label may hide valuable insights into annotator disagreement, particularly in tasks where subjectivity plays a crucial role. In this work, we explore methods for identifying subjectivity in recognizing the human values that motivate arguments. We evaluate two main approaches: inferring subjectivity through value prediction vs. directly identifying subjectivity. Our experiments show that direct subjectivity identification significantly improves the model performance of flagging subjective arguments. Furthermore, combining contrastive loss with binary cross-entropy loss does not improve performance but reduces the dependency on per-label subjectivity. Our proposed methods can help identify arguments that individuals may interpret differently, fostering a more nuanced annotation process 1 1 1 Our code is publicly available at https://github.com/Amir-Homayouni/subjectivity-value . 1 Introduction Human values, spanning concepts such as benevolence and self-determination, are the motivations that guide our choice and action, and are ordered by importance to form a system of value priorities Schwartz 1994 . Value-laden arguments are statements grounded in our personal values, which we use to motivate our choices Bench-Capon 2003 . The identification of the values that support our arguments can reveal our deepest motivations, and as such has been recently investigated in the NLP community Kiesel et al. 2023 ; Kiesel et al. 2024 . Supervised NLP methods have been proposed to identify the values that support a text segment Kiesel et al. 2022 ; Liscio et al. 2022 . Typically, the ground truth labels are chosen through majority aggregation of the annotations Hoover et al. 2020 ; Kiesel et al. 2023 or the annotators engage in discussions to reach an agreement on the annotation Liscio et al. 2021 ; Lei et al. 2024 . However, due to the subjective nature of valuing Mackie 1988 ; Stroud 1988 , disagreement in the interpretation of the values that support an argument is natural. For example, consider the following argument in favor of a multi-party political system: ‘‘(it) would bring many new and fresh ideas into the forefront’’. Alice may associate the argument with the value of universalism since a multi-party system can provide all people with equal opportunities. Bob, instead, may connect it to the values of achievement and personal security, because a multi-party system can more effectively address issues than a single-party system and thus provide more security to its citizens 2 2 2 The argument and the two annotations are sourced from the ValueEval’23 Shared Task Kiesel et al. 2023 . . Approaches relying on consensus or majority aggregation might obscure the inherent subjectivity of the identification of the values behind arguments, leading to misinterpretation or, at worst, promotion of biases stemming from the annotation process. Identifying subjectivity in value-laden arguments thus has broader implications, particularly in contexts that demand participatory deliberation and collective decision-making. The assumption that annotations should be aggregated into a unique label is being questioned within the NLP community Röttger et al. 2021 ; Weerasooriya et al. 2023 ; van der Meer et al. 2024 , in applications ranging from hate speech detection Kocoń et al. 2021 ; Mostafazadeh Davani et al. 2022 to sentiment and emotion detection Deng et al. 2023 . In line with these works, in this paper we propose methods to identify subjective value-laden arguments—that is, detect arguments where annotators may have different interpretations of the values that support them. We envision two primary applications of this work. (1) Identifying arguments that might be subjective can support a more nuanced annotation process by prompting the collection of additional annotations. (2) Identifying the arguments that might lead to misinterpretation and divergent views during participatory deliberations can prompt moderators to ask additional questions that could promote self or inter-participant reflection. Contribution We propose two different approaches to detecting subjectivity in value identification within discourse: three methods for inferring subjectivity from individual value annotations, and three methods for directly identifying subjectivity, which we further enhance with contrastive learning strategies. We validate the approaches on the Touché23-ValueEval dataset Kiesel et al. 2023 . Our results show that we can identify subjectivity in value-laden arguments, and that directly identifying subjectivity—rather than inferring it through value prediction—greatly improves performance. Additionally, leveraging contrastive loss does not improve subjectivity prediction performance but brings other advantages. 2 Related works We review related works on identifying values in text and on subjectivity in NLP applications. 2.1 Identifying Values in Text Identifying the value(s) that support a natural language statement has been approached through word count and sentence embedding similarity to dictionaries of value-laden words Araque et al. 2020 . More recent approaches employ supervised machine learning on annotated datasets Alshomary et al. 2022 ; Huang et al. 2022 ; Park et al. 2024 ; Liscio et al. 2025 ; Senthilkumar et al. 2024 . In particular, Kiesel et al. 2022 focuses on identifying and classifying the values underlying arguments. They successfully fine-tuned a BERT model Devlin et al. 2019 on multi-cultural arguments, and later extended the dataset for the ValueEval challenge at SemEval’23 Kiesel et al. 2023 . In this study, unlike the previous work that uses golden ground truths to identify values, we utilize a dataset consisting of annotation of values to arguments to explore the subjectivity of annotations in recognizing the values behind arguments. 2.2 Identifying Subjectivity Subjectivity is playing an increasingly central role in various NLP tasks Plank 2022 . Datasets that report individual-level annotation (e.g., Aroyo et al. 2023 ) facilitate the modeling of individual and group annotation behavior, with different annotation paradigms shown to have a great impact on data quality and model performance Röttger et al. 2021 . This information allows to represent ground truth as a label distribution, preserving diverse human judgments and minority opinions Weerasooriya et al. 2023 . Different methods account for subjectivity by combining annotator and annotation embeddings Deng et al. 2023 , modeling multi-annotator architectures Mostafazadeh Davani et al. 2022 , and capturing annotators’ perspectives by combining their demographic information and their opinions on online content Fleisig et al. 2023 . Demographic information of annotators has been employed as a feature, however with mixed results Goyal et al. 2022 ; Wan et al. 2023 ; Orlikowski et al. 2023 . Subjectivity has also been explored as part of the sampling strategy of Active Learning methods, e.g. to select the next sample to be annotated Baumler et al. 2023 ; Wang and Plank 2023 or the next annotator that should annotate it van der Meer et al. 2024 . These approaches primarily focus on predicting the opinions of individuals or groups of annotators. Instead, our methods identify whether we can expect disagreement among annotators identifying the values that support an argument, thus not being reliant on specific individuals or groups. 2.3 Measuring disagreement on moral ambiguity In morally ambiguous scenarios, disagreements naturally arise due to the inherent diversity and pluralism of values people hold. Grounding theories like value pluralism suggest that such disagreements reflect legitimate differences in moral frameworks rather than mere errors Kekes 1996 . Among others, the Moral Foundations Theory illustrates how different moral intuitions (care, fairness, authority, loyalty, sanctity) lead to divergent judgments among groups such