Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks

Navya Mehrotra ⋆, Adam Visokay ⋆, Kristina Gligorić · 2026

Read Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks by Navya Mehrotra ⋆, Adam Visokay ⋆, Kristina Gligorić (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21404.txt.

Introduction

Mehrotra and co-authors address a flaw in using language models to label text. Model labels reflect some human groups better than others, and standard fixes assume one true label. For subjective tasks, group disagreement itself matters. Their Perspective-Driven Inference method estimates the spread of labels across groups with a small human labeling budget. A sampling plan targets groups the model proxies worst. Tests on politeness and offensiveness ratings show gains for those groups.

Catalog record

CREATORNavya Mehrotra ⋆, Adam Visokay ⋆, Kristina Gligorić
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48677
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21404

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks Navya Mehrotra ⋆, Adam Visokay ⋆, Kristina Gligorić Abstract Large language models are increasingly used to annotate texts, but their outputs reflect some human perspectives better than others. Existing methods for correcting LLM annotation error assume a single ground truth. However, this assumption fails in subjective tasks where disagreement across demographic groups is meaningful. Here we introduce Perspective-Driven Inference , a method that treats the distribution of annotations across groups as the quantity of interest, and estimates it using a small human annotation budget. We contribute an adaptive sampling strategy that concentrates human annotation effort on groups where LLM proxies are least accurate. We evaluate on politeness and offensiveness rating tasks, showing targeted improvements for harder-to-model demographic groups relative to uniform sampling baselines, while maintaining coverage. † † ⋆ Equal contribution. 1 Introduction LLMs are promising tools for reducing annotation burden in computational social science (CSS) 61 ; 28 . Yet, LLM annotations can be wrong 11 , encode particular positionalities 39 , and reproduce stereotypes 43 . Methods like Prediction-Powered Inference 3 and design-based supervised learning 13 offer a repair 21 , addressing these limitations by combining potentially inaccurate LLM proxies with a small set of ground truth human labels, yielding valid downstream inference even when the LLM proxy is imperfect. Figure 1: Overview of the Perspective-Driven Inference . Starting from a corpus of n texts, we collect LLM annotations, initialize human annotation via uniform sampling, and then enter an adaptive loop that predicts LLM error from demographic features, sampling human annotations across groups. The core idea is to prioritize human annotation for texts where LLMs poorly represent the target demographic group. Once the annotation budget is exhausted, we apply inverse probability weighting (IPW) to produce group-level estimates \theta^{*} . However, these repair methods assume a ground truth exists. In many CSS settings, ground truth does not exist. Reasonable people disagree about whether language is toxic, empathetic, or polite, and that disagreement is not error to be corrected, but a signal to be preserved. Collapsing annotator disagreement into a majority label discards the very variation that matters for understanding how people use and perceive language. A common method to address is persona prompting 48 : conditioning LLMs on demographic attributes to simulate the perspectives of specific groups. However, persona prompting does not always improve performance of large language models 60 ; 46 : it can exaggerate group differences 10 , misrepresent minority viewpoints 52 , and in many settings performs no better than a generic prompt 60 ; 46 ; 57 . Here, we introduce Perspective-Driven Inference , a framework for valid corpus inference in subjective annotation tasks where no single ground truth exists (illustrated in Figure 1 ). Our goal is not to train LLMs to make more accurate predictions, but to leverage their annotations to conduct robust downstream statistical inference that faithfully retains annotator disagreement. Our contributions are: 1. Framework. We formalize multi-perspective corpus inference as estimation of a vector estimand (one entry per demographic group), rather than a single scalar, and show how standard PPI-based correction extends to this setting. 2. Sampling method. We develop a sampling strategy that concentrates the human annotation budget on demographic groups for which LLM annotations are predicted to be less reliable, with the goal of improving inference for underrepresented or poorly modeled perspectives. 3. Evaluation. We evaluate Perspective-Driven Inference on three datasets: politeness ratings, offensiveness ratings, and synthetic binary annotations, finding improved accuracy for targeted demographic groups, while maintaining coverage. 2 Background Annotators’ background A growing body of NLP literature views annotation disagreement not as noise, but valuable signal 50 ; 20 . Prior work has shown considerable heterogeneity in annotator preferences on subjective tasks, ranging from toxicity 40 and sexism detection 47 , to theory of mind modeling 14 and counterspeech generation 31 , with demographic axes like gender 8 and race 25 shifting annotations systematically 1 . Modeling this disagreement explicitly, rather than collapsing it into majority vote, has become a concern in NLP 18 ; 57 ; 56 . Recent efforts push toward characterizing how demographic factors shape human preferences in LLM outputs 30 ; 54 ; 9 , and using that understanding to build systems that represent multiple perspectives 44 ; 16 ; 35 . What is different about our work is that we ask: how can we efficiently leverage limited budgets of human input, given that models do not capture different perspectives equally well 42 ; 38 ? In doing so, our goal is not to use LLMs to make better predictions, but to use LLM annotations as inputs to downstream analyses, while preserving diverse perspectives on subjective tasks. Valid inference with proxy annotations Corpus-level inference from text has roots in literary analysis 58 and causal inference in NLP 15 . Early work formalized the task of inferring document class prevalence under uncertainty 22 . A separate thread examined whether annotations themselves are valid 51 ; 37 , and how annotation error and genuine label variation can be disentangled 55 ; 6 . More recent work in CSS turns to downstream inference under annotation imperfection, using LLM outputs as surrogates for human labels and developing statistically rigorous methods for correction 53 ; 13 ; 2 . We contribute to this line of work by complicating the assumption that a ground truth exists at all: in subjective tasks, disagreement is constitutive, and we show how these repair methods apply in settings where no single correct label can be defined, and the target estimand is a vector, rather than a scalar. Adaptive sampling Active learning methods reduce annotation cost by selecting the most informative instances for labeling 5 ; 4 , with established tools to support these workflows 27 ; 49 ; 32 . Work in the multi-annotator setting specifically targets instances where annotators are likely to disagree 7 , and recent methods allocate annotation effort between humans and LLMs based on uncertainty 26 ; 36 ; 59 . Most related to our approach is sampling based on LLM annotation confidence 19 , and stratified PPI 17 , which improves inference efficiency by partitioning samples into strata. In contrast, our stratification is along perspectives rather than strata with data subsets: we sample human annotators to cover the distribution of viewpoints, not to reduce label uncertainty around a single ground truth. 3 Methods 3.1 Problem setup We have a corpus of n independent and identically distributed (i.i.d.) text instances T_{1},\dots,T_{n} . We consider a perspective-aware setup: each instance is annotated by multiple annotators, each coming from a particular perspective. Specifically, each observation is a (text, annotator) pair (T_{i},d_{i}) , where d_{i}\in\mathcal{D} encodes the demographic profile of the annotator (e.g., gender, race, age, education level). The human annotation H_{i} therefore captures the judgment of an annotator with demographics d_{i} on text T_{i} . Note that we focus on demographic features as a proof of concept. In principle, d_{i} features could include any axis along which human perspectives vary, informed by theoretical background knowledge, or exploratory analyses. We partition annotators into K demographic groups \mathcal{G}=\{g_{1},\dots,g_{K}\} . We wish to estimate a vector of group-specific quantities \theta^{*}=(\theta^{*}_{g_{1}},\dots,\theta^{*}_{g_{K}}), where each \theta^{*}_{g_{k}}=\mathbb{E}[H_{i}\mid d_{i}\in g_{k}]

Read the complete text