An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration

Maja Pavlovic, Silviu Paun, Massimo Poesio · 2026

Read An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration by Maja Pavlovic, Silviu Paun, Massimo Poesio (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18648.txt.

Introduction

Maja Pavlovic, Silviu Paun, and Massimo Poesio audit, in a 2026 report, how human soft labels shape learning and calibration. Past work mixed two effects: the gain from soft labels and the silent fix of mislabeled data. By re-annotating subsets of MNIST and a synthetic variant, they separate the two. Human soft labels give small accuracy gains, but their larger value is regularization: better calibration on hard samples and steadier convergence across runs. Models trained this way mirror human uncertainty, while synthetic-label models fail to align with people.

Catalog record

CREATORMaja Pavlovic, Silviu Paun, Massimo Poesio
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67140
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18648

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration Maja Pavlovic, Silviu Paun, Massimo Poesio Abstract Central to human-aligned AI is understanding the benefits of human-elicited labels over synthetic alternatives. While human soft-labels improve calibration by capturing uncertainty, prior studies conflate these benefits with the implicit correction of mislabeled data (mode shifts), obscuring true effects of soft-labels. We present a controlled audit of soft-label learning across MNIST and a synthetic variant, re-annotating subsets to extract human uncertainty. By decoupling soft-label supervision from underlying label mode shifts, we show that while human soft-labels do provide accuracy gains, their larger value lies in acting as a regularizer that improves model calibration on difficult samples and promotes stable convergence across training runs. Dataset cartography reveals models trained on human soft-labels mirror human uncertainty, whereas those trained on synthetic labels fail to align with humans. Broadly, this work provides a diagnostic testbed for human-AI uncertainty alignment. 1 Introduction Model calibration ensures that predicted probabilities accurately reflect the true likelihood of outcomes. Traditionally, models are trained and evaluated using a one-hot encoded vectors. Within this paradigm, smaller models have been shown to produce well-calibrated predictions [ 32 ] ; however, as model complexity increases, models often become less calibrated [ 16 , 1 ] . While early approaches demonstrate that soft-labels enhance calibration [ 30 ] and generalization [ 17 ] , recent work suggests that synthetically generated data can further refine calibration by exposing models to a wider variety of edge cases [ 43 , 7 , 4 ] . However, using teacher model soft-labels [ 17 ] or otherwise algorithmically generated soft-labels risks propagating synthetic biases rather than reflecting the nuances of human uncertainty. Although such targets provide a softer signal than one-hot vectors, recent analyses find why algorithmic soft-labels can actually harm uncertainty ranking [ 49 , 46 ] . In contrast to these synthetic soft-labels, human-elicited soft-labels encode the diverse origins of uncertainty that characterize real-world data and have been shown to directly improve both model calibration and robustness [ 37 , 10 ] . Human uncertainty is multifaceted, originating from varied interpretations of annotation schemes [ 35 , 44 ] , differences in cultural or background knowledge [ 39 , 35 ] , cognitive difficulties in separating similar concepts [ 11 , 44 ] , or data-specific issues like image occlusion and noise [ 37 , 40 , 11 ] . Additionally, difficulty arises when instances legitimately belong to multiple categories [ 6 , 39 ] , or are inherently vague, allowing for multiple valid interpretations due to ambiguity rather than overlapping categories [ 25 , 40 ] . Recent efforts have begun integrating individual human uncertainty into annotated datasets [ 3 , 35 , 10 ] . While the utility of soft-labels is thought to depend on their alignment with human label variation [ 38 , 13 , 44 ] , our fundamental understanding of how and why human-derived labels influence model training dynamics remains limited. Prior studies demonstrating accuracy gains from human soft-labels [ 37 ] conflate the soft-supervision with the implicit correction of the underlying label mode (i.e. shifting the argmax to a more accurate class during re-annotation). Decoupling these effects requires a controlled testbed, akin to the use of MNIST to validate hypotheses in generalization [ 47 ] or challenge assumptions regarding adversarial robustness [ 42 ] . To provide a similarly principled investigation into human soft-labels, we conduct experiments on two digit datasets (MNIST and a synthetic equivalent), re-annotating subsets to explicitly capture individual human class uncertainty. Within these testbeds, we restrict model capacity across three tractable baselines to capture fundamental learning differences. By evaluating these models across strata of human agreement and comparing soft-labels to standard training paradigms, we offer the following contributions: • An audit of two fundamental datasets with respect to human-centric soft-labels: Specifically, we demonstrate that a substantial share of the accuracy gains on MNIST can be attributed to a shift in the label mode during re-annotation. Furthermore, re-annotation of the synthetic digits reveals that model-generated labels in our testbed fail to align with human visual intuition across much of the dataset; consequently, we observe that models easily fit these synthetic labels but struggle to model human uncertainty. • Refining the soft-label advantage: We provide a thorough evaluation of human soft-labels by disentangling their benefits from mode shifts during re-labeling; a control for isolating true effects absent in previous evaluations. While we confirm accuracy gains, we show that the benefit of human uncertainty lies in improving model calibration particularly across difficult samples and facilitating more stable convergence across runs. We further corroborate that these gains hold even when using an order of magnitude fewer labels per image than prior work. • Aligning training dynamics with human uncertainty: We find that models supervised by human soft-labels mirror human uncertainty throughout the learning process; leading to models whose internal representations are more aligned with human-centric uncertainty than those trained on standard one-hot labels, and notably more than those trained on synthetic labels. 2 Dataset & annotation scheme 2.1 Dataset setup By deliberately using simple, well-defined digit datasets, we create a controlled environment to evaluate how human soft-labels influence model performance without the confounding variables of natural imagery. Dataset selection: To investigate model behavior across the full spectrum of sample difficulty, we evaluate four digit-based datasets: the standard MNIST [ 24 ] , the Swedish historical handwritten dataset ARDIS [ 23 ] , and two synthetic variants of Ambiguous-MNIST [ 29 , 45 ] . Standard datasets such as MNIST are commonly dominated by ’easy’ instances that lead to rapid loss convergence, however, work on annotator uncertainty [ 37 ] and dataset cartography [ 41 ] suggests that a model’s ability to generalize is more closely tied to its performance on ’ambiguous’ and ’hard’ instances. Rather than relying on the skewed distribution of the original sources, we construct datasets balanced across the difficulty spectrum. Specifically, we employ Dataset Cartography [ 41 ] as a filtering mechanism to identify high-information candidates for human labeling across easy, ambiguous, and hard regions. We adopt this over standard confidence-based sampling, as Data Maps have been shown to outperform simple uncertainty metrics in active learning and generalization contexts [ 48 ] . Dataset construction: To construct this corpus, we first process all four datasets by removing duplicate images across the corpora. Notably, this de-duplication mitigates the train-test leakage we observed in three of the four candidate datasets [ 23 , 45 , 29 ] , ensuring a cleaner signal for evaluating true generalization. Following de-duplication, we perform cartography mapping using a simple one-hidden-layer feed-forward network (see Appendix A.2 ). Guided by these steps, we select MNIST and Mukhoti’s [ 29 ] distributional Ambiguous-MNIST as our primary sources. MNIST is chosen over ARDIS because its larger scale (70k vs 7,474 images) provides a richer tail of naturally occurring hard and ambiguous cases. Between the synthetic datasets, [ 29 ] offers a more comprehensive distribution across the difficulty spectrum and better adheres to the natural data manifold compared to [ 45 ] (see Appendix A ). Difficulty regions are defined by thresholds on confidence ( \mu ) and

Read the complete text