Bridging the Gap: In-Context Learning for Modeling Human Disagreement

Benedetta Muscato, Yue Li, Gizem Gezici, Zhixue Zhao, Fosca Giannotti · 2025

Read Bridging the Gap: In-Context Learning for Modeling Human Disagreement by Benedetta Muscato, Yue Li, Gizem Gezici, Zhixue Zhao, Fosca Giannotti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06113.txt.

Introduction

Labeling text for hate or offense splits human judges, and averaging their votes erases real perspectives. Benedetta Muscato and co-authors ask whether language models can hold the disagreement instead. They test four open models with in-context learning, feeding labels three ways: merged hard labels, and separate hard or soft labels that keep each annotator visible.

Zero-shot prompting could voice multiple perspectives. Few-shot examples, oddly, often narrowed the range instead of widening it. How examples were chosen, by similarity or by disagreement, mattered more than their order. The 2025 study warns that teaching models to mimic majorities is easy, and teaching them pluralism is not.

Catalog record

CREATORBenedetta Muscato, Yue Li, Gizem Gezici, Zhixue Zhao, Fosca Giannotti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-06113

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Bridging the Gap: In-Context Learning for Modeling Human Disagreement Benedetta Muscato, Yue Li, Gizem Gezici, Zhixue Zhao, Fosca Giannotti Abstract Large Language Models (LLMs) have shown strong performance on NLP classification tasks. However, they typically rely on aggregated labels—often via majority voting—which can obscure the human disagreement inherent in subjective annotations. This study examines whether LLMs can capture multiple perspectives and reflect annotator disagreement in subjective tasks such as hate speech and offensive language detection. We use in-context learning (ICL) in zero-shot and few-shot settings, evaluating four open-source LLMs across three label modeling strategies: aggregated hard labels, and disaggregated hard and soft labels. In few-shot prompting, we assess demonstration selection methods based on textual similarity (BM25, PLM-based), annotation disagreement (entropy), a combined ranking, and example ordering strategies (random vs. curriculum-based). Results show that multi-perspective generation is viable in zero-shot settings, while few-shot setups often fail to capture the full spectrum of human judgments. Prompt design and demonstration selection notably affect performance, though example ordering has limited impact. These findings highlight the challenges of modeling subjectivity with LLMs and the importance of building more perspective-aware, socially intelligent models. Warning: This paper contains examples that may be offensive or upsetting. Figure 1: Overview of our findings. Our analysis shows that LLMs can effectively reflect multiple perspectives in hate speech detection, when prompted to consider diverse viewpoints or when predicting aggregated labels. This is illustrated using Jensen-Shannon Divergence (JSD) on a representative annotation example. 1 Introduction In traditional classification tasks, annotations are typically collected from multiple annotators, often through crowdsourcing, and consolidated into a single ground truth label per example. While this method works well when annotators are in full agreement and the aggregated label reflects a clear consensus, it fails to account for cases where multiple, equally valid interpretations are possible. This shortcoming is especially pronounced in subjective tasks 1 1 1 In Natural Language Processing (NLP), subjective tasks are those with multiple valid interpretations, influenced by personal perspectives, emotions, or social context Röttger et al. 2022 . where inherent annotator disagreement can emerge for valid reasons such as text ambiguity, genuine disagreement, subjective preferences, and two or more plausible perspectives Plank et al. 2014 . For instance, hate speech detection often entails contentious annotations, as individuals may interpret hateful content differently based on their personal experiences or cultural backgrounds Davani et al. 2021 ; Akhtar et al. 2021 . Similarly, the detection of offensive and abusive language is highly dependent on subjective judgments, making the task even more complex Del Arco et al. 2021 ; Husain and Uzuner 2021 . A new line of research highlights that leveraging disagreements in human-annotated datasets can enhance both model performance and confidence Casola et al. 2023 ; Davani et al. 2022 ; Sandri et al. 2023 ; Muscato et al. 2024 . This emerging direction, known as Perspectivism Basile et al. 2021 , suggests a paradigm shift in model design philosophy, advocating for the development of systems that are not only Perspective-aware but also more Responsible and Socially Intelligent Liu et al. 2023 ; Kovač et al. 2023 . Nevertheless, LLMs, including systems such as GPT-series, exhibit notable limitations in modeling the diversity of user preferences Pavlovic and Poesio 2024 ; Feng et al. 2024 across different sociodemographic groups Wang et al. 2025 . Srivastava et al. 2022 show that LLMs are prone to inherent biases, which are particularly pronounced in ambiguous contexts where human judgments are subjective. Similarly, Santurkar et al. 2023 highlight that LLMs tend to reflect a predominantly left-leaning perspective, further limiting their ability to provide diverse opinions. In this paper, we focus on the interplay between in-context learning (ICL) and human disagreement in subjective NLP tasks. Specifically, we investigate whether multi-perspective ICL can serve as an effective and practical alternative to fine-tuning for capturing diverse perspectives , especially with limited annotated data and computational resources. Through a comprehensive analysis of established ICL methods, we evaluate their performance in both zero-shot and few-shot prompting. Our study explores thirty different combinations across four LLMs , three subjective tasks (hate speech, offensive language, and abusive language detection) in Section 3.1 , three types of label spaces (aggregated hard and disaggregated hard & soft) in Section 3.2 , and five demonstration example strategies in Section 3.3 . These include both selection methods (textual similarity, annotator disagreement, and two-stage ranking) and ordering techniques (random and In-Context Curriculum Learning (ICCL) Liu et al. 2024 ). Our multi-perspective ICL approach, which specifically instructs the model to consider different viewpoints, is compared to the standard prompting (baseline). Although our findings demonstrate that the multi-perspective approach performs better than the baseline in predicting aggregated labels, it struggles to capture subjective nuances in disaggregated labels. Our findings highlight the need for perspective-aware LLMs to mitigate polarization and societal bias, as shown in Figure 1 , and underscores ongoing challenges in applying ICL to subjective tasks. The closest work to our methodology is Pavlovic and Poesio 2024 , which uses a single closed-model LLM (GPT-3.5-turbo) to compare human and LLM opinion distributions leveraging zero-shot ICL with role-playing, generating only disaggregated soft labels in both our English benchmark datasets, along with an additional Arabic one. We extend prior work in the following ways: 1) using four open-source LLMs instead of a single closed model, 2) prompting the models in both zero-shot and few-shot settings, 3) incorporating tailored selection and ordering strategies for demonstration organization in few-shot prompting, specifically designed for subjective tasks and 4) expanding the label space with: aggregated hard labels, disaggregated hard & soft labels. Thus, we posit that our results provide generalizable evidence on the limitations and potential of LLMs to handle subjective disagreement through ICL. 2 Background 2.1 Socially Intelligent and Responsible LLMs An AI system is deemed socially intelligent when it can understand and reason about the intentions, emotions, and mental states of others Sap et al. 2022 ; Pereira et al. 2016 . Beyond basic functionality, these models or agents aim to sense, perceive, reason about, learn from, and respond to the affective behavior and cognition of other agents (whether human or artificial) Qiu et al. 2022 . Like humans, they evolve their perspectives through interaction, integrating subjective preferences and beliefs Mittelstädt et al. 2024 ; Kirk et al. 2025 . In the broader context of Responsible AI, which aims not just at better performance, but at delivering societal benefit, social intelligence helps align LLMs with diverse user preferences Mathur et al. 2024 , reducing harm and fostering pluralistic alignment across individual and collective needs Sorensen et al. 2024 ; Tahaei et al. 2023 ; Kirk et al. 2024 . 2.2 Learning from Human Disagreement Human disagreement, often seen as noise that harms label quality in crowd-sourcing Artstein 2017 ; Fleisig et al. 2024 , is increasingly reframed as a form of plausible human label variation (HLV) Plank 2022 , especially in subjective tasks e.g., hate speech, stance, and emotion detection, where no single ground truth label may exist. While traditional methods rely on hard (discrete) aggregated labels, such as those consolidated through majority voting, the perspectivist literature Frenda et al. 2024 preserves disagreement using either hard or soft (continuous) disaggregated labels. Recent research shows that

Read the complete text