Subjective Logic Encodings
Jake Vasilakes, Chrysoula Zerva, Sophia Ananiadou · 2025
Read Subjective Logic Encodings by Jake Vasilakes, Chrysoula Zerva, Sophia Ananiadou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12225.txt.
Introduction
Jake Vasilakes and co-authors build a new way to teach models from labeled data when the labelers disagree. Most pipelines treat disagreement as noise and erase it through cleanup or majority votes. On tasks like sentiment or hate speech, though, disagreement is natural and carries real information.
Their approach draws on subjective logic, a formalism for opinions that hold belief, doubt, and uncertainty at once. Labels become encoded opinions rather than flat facts, keeping each annotator's stance in the training signal. The method widens the data perspectivism program, which had leaned on disagreement counts alone. Models can then learn the uncertainty itself, not just an averaged answer.
Catalog record
| CREATOR | Jake Vasilakes, Chrysoula Zerva, Sophia Ananiadou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12225
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Subjective Logic Encodings Jake Vasilakes, Chrysoula Zerva, Sophia Ananiadou Abstract Many existing approaches for learning from labeled data assume the existence of gold-standard labels. According to these approaches, inter-annotator disagreement is seen as noise to be removed, either through refinement of annotation guidelines, label adjudication, or label filtering. However, annotator disagreement can rarely be totally eradicated, especially on more subjective tasks such as sentiment analysis or hate speech detection where disagreement is natural. Therefore, a new approach to learning from labeled data, called data perspectivism, seeks to leverage inter-annotator disagreement to learn models that stay true to the inherent uncertainty of the task by treating annotations as opinions of the annotators, rather than gold-standard facts. Despite this conceptual grounding, existing methods under data perspectivism are limited to using disagreement as the sole source of annotation uncertainty. To expand the possibilities of data perspectivism, we introduce Subjective Logic Encodings (SLEs), a flexible framework for constructing classification targets that explicitly encodes annotations as opinions of the annotators. Based on Subjective Logic Theory, SLEs encode labels as Dirichlet distributions and provide principled methods for encoding and aggregating various types of annotation uncertainty —annotator confidence, reliability, and disagreement— into the targets. We show that SLEs are a generalization of other types of label encodings as well as how to estimate models to predict SLEs using a distribution matching objective. We make our code publicly available at https://github.com/jvasilakes/SLEncodings . 1 Introduction Machine learning is inherently biased. Hovy & Prabhumoye (2021) identify five sources of bias: data selection, annotation, input representations, model, and experimental design. While all five have been studied in detail in previous works, a new view on annotation bias called data perspectivism ( Basile et al., 2021 ) has gained traction. Data perspectivism re-frames annotations as opinions of the annotators rather than “gold-standard” facts. It also stands in contrast to existing methods for learning from noisy labels, which attempt to eradicate noise rather than embrace it ( Song et al., 2022 ) . This movement is inspired by the myth that there ought to be a single true label and that annotator disagreement is to be avoided ( Aroyo & Welty, 2015 ) . Data perspectivism thus aims to embrace and leverage annotation uncertainty to build ML systems that are more representative of the event being modeled. 1.1 Contributions Despite this conceptual grounding, existing methods under data perspectivism focus on utilizing only one of many types of annotation uncertainty —annotator disagreement— resulting incomplete opinion representations. To expand on this, we propose Subjective Logic Encodings (SLE), a flexible framework for constructing target distributions for classification tasks which explicitly encodes annotations as subjective opinions of the annotators. SLEs are based on Subjective Logic Theory ( Jøsang, 2016 ) , which provides principled methods for encoding and aggregating various types of annotation uncertainty —annotator disagreement, as well as reliability and subjective uncertainty— into target Dirichlet distributions. We estimate a neural network to predict these distributions using a simple distribution matching objective, and show that this is equivalent to standard cross-entropy training with gold-standard labels in the absence of uncertainty. Experiments with both synthetic and real datasets in natural language processing and computer vision show that SLEs are more general and flexible method for learning from annotation uncertainty while matching the performance of existing methods. 2 Sources of Uncertainty in Annotation We are interested in modeling aleatoric uncertainty , also called data uncertainty, which refers to the irreducible uncertainty inherent in the data or events being modeled. It stands in contrast to epistemic uncertainty —due to a lack of knowledge regarding the events— and distributional uncertainty —uncertainty regarding out-of-distribution (OOD) events. Generally, aleatoric uncertainty arises from noise inherent in the event being modeled (e.g., a dice roll) or from measurement noise ( Gal, 2016 ) . More specifically, we view annotation as a measurement process and break it down into three indicators of uncertainty: inter-annotator disagreement indicates inherent event uncertainty, while annotator reliability and confidence indicate measurement uncertainty. We discuss each of these indicators below. Annotator Reliability: An annotator may be highly confident in their annotations but still often provide incorrect or inconsistent labels due to a lack of expertise, the difficulty of the annotation task, etc. Conversely, adversarial annotators may purposefully provide poor annotations. Annotator reliability is thus an indicator of measurement noise. Simple measures of reliability are average agreement between annotators ( Tratz & Hovy, 2010 ) , average Cohen’s \kappa between pairs of annotators ( Hovy et al., 2013 ) , and the G-index ( Gwet, 2008 ) . Previous works have also learned models of annotator reliability directly from data ( Hovy et al., 2013 ; Jagabathula et al., 2017 ; Li et al., 2019 ) . Annotator Confidence: Borderline examples, difficult annotation tasks, or unclear annotation guidelines can all influence an annotator’s uncertainty in their assigned labels. Thus individual annotator confidence is an indicator of measurement noise. Previous works have measured confidence by asking annotators additional questions ( Nguyen et al., 2014 ) or eliciting probabilistic labels ( Collins et al., 2022 ) . Inter-Annotator Disagreement: Even assuming perfect annotator reliability and confidence, multiple annotators may still assign different labels to the same event, indicating an inherent uncertainty in the even ( Plank et al., 2014 ) . Inter-annotator disagreement is thus an indicator of aleatoric uncertainty. There has been a recent surge of interest in leveraging disagreements between annotators for improving classification (see Uma et al. (2021) for a survey). It is commonly assumed that crowd annotations are samples from the same categorical distribution over labels for a given example (e.g., this assumption is explicitly stated in Baan et al. (2022) ). The existence of annotator confidence and reliability, however, suggest that such samples pass through a intermediate “measurement” process, i.e., the subjective opinion of the annotator, which lends uncertainty to each annotation. It is therefore necessary to explicitly encode annotations as opinions —i.e., including information regarding annotator confidence and reliability. In a more general sense, we ought to encode both first-order annotation uncertainty (i.e., probability of each label in a categorical distribution) as well as second-order uncertainty (i.e., uncertainty regarding the probabilities themselves). The next section discusses how to do this using Subjective Logic. 3 Background: Subjective Logic Theory The above shows that it is natural to view annotations as subjective opinions of the annotators. However, the second-order uncertainty we aim to encode is incompatible with the current status quo of vector representations, which are limited to first order uncertainty. Subjective Logic (SL) is a type of probabilistic logic that explicitly encodes events as the opinions of agents, with separate dimensions for first- and second-order uncertainty. We here provide a brief overview of the aspects of SL necessary to understand SLEs. 3.0.1 Opinion Representation In SL, a subjective opinion regarding some item i according to an agent m over a domain of K possible events (e.g., class labels) is denoted \omega^{(i)}_{m}=(\bm{b}^{(i)}_{m},u^{(i)}_{m},\bm{a}) , where \bm{b}^{(i)}_{m}\in[0,1]^{K} is a vector of beliefs, u^{(i)}_{m}\in[0,1] represents the uncertainty of the opinion, and \bm{a}\in[0,1]^{K} is a vector of base rates or prior probabilities over the event space 1 1 1 Without loss of generality, we assume uniform priors throughout this work. . These parameters are subject