REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations
Sajjad Ghiasvand, Mark Beliaev, Mahnoosh Alizadeh, & Ramtin Pedarsani · 2026
Read REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations by Sajjad Ghiasvand, Mark Beliaev, Mahnoosh Alizadeh, & Ramtin Pedarsani (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-17289.txt.
Introduction
Ghiasvand, Beliaev, Alizadeh, and Pedarsani tackle noisy crowd labels in language-model training. Standard vote-counting lets careless annotators corrupt the model. Their method REALM learns a reliability score for each annotator alongside the model itself. Each label is treated as a mix of the model's guess and random chance. It topped rivals on real crowdsourced text tasks and beat plain noisy training in 152 of 162 question-answering setups.
Catalog record
| CREATOR | Sajjad Ghiasvand, Mark Beliaev, Mahnoosh Alizadeh, & Ramtin Pedarsani |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54751 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.17289
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
REALM: Reliable Expertise-Aware Language Model Fine-Tuning from Noisy Annotations Sajjad Ghiasvand, Mark Beliaev, Mahnoosh Alizadeh, & Ramtin Pedarsani Abstract Supervised fine-tuning of large language models relies on human-annotated data, yet annotation pipelines routinely involve multiple crowdworkers of heterogeneous expertise. Standard practice aggregates labels via majority vote or simple averaging, discarding annotator identity and causing the model to absorb the errors of unreliable annotators directly into its parameters. We propose REALM, a method that jointly learns the model parameters and a scalar expertise value for each annotator entirely unsupervised, requiring no supervision beyond annotator identity. The key idea is to model each observed label as a mixture between the model’s prediction and a uniform random guess, weighted by the annotator’s learned expertise. We extend REALM to a multi-task setting via a learned expertise matrix that captures per-annotator reliability across tasks. We evaluate on five question answering benchmarks, fine-tuning three sizes of Flan-T5 under simulated noisy annotations. The proposed algorithm consistently outperforms the naive noisy SFT in the large majority of single- and multi-task settings, across datasets, model sizes, and noise types, with accuracy improvements of up to 50 % 50\% in the most adversarial regime and gains that grow with model capacity. 1 Introduction Supervised fine-tuning (SFT) on human-annotated data is a cornerstone of modern language model development [ 19 , 6 , 34 , 1 ] . In practice, annotation pipelines rely on multiple crowdworkers who vary widely in domain expertise, task familiarity, and attention [ 19 , 1 ] . Despite this, the overwhelming majority of fine-tuning pipelines treat all annotations as equally reliable, aggregating labels via majority vote or simple averaging before training [ 9 , 23 ] . This discards information about annotator identity and quality, and causes the model to absorb the errors of unreliable annotators directly into its parameters. The problem is particularly acute in the low-budget crowdsourcing setting, where a small number of annotators with heterogeneous expertise label large quantities of data. When one annotator is highly reliable and two are near-random, a model trained naively on the combined annotations is learning from noise the majority of the time. Recent work has shown that annotation quality matters far more than quantity for SFT [ 33 ] , yet existing methods for handling label noise [ 18 , 13 , 20 ] typically assume a fixed noise transition matrix and require either a clean validation set or strong assumptions about the noise structure — neither of which is available in the crowdsourcing setting. The question we address is whether the model can instead identify and down-weight unreliable annotators automatically, using only the observed labels and annotator identities, without any additional supervision on annotator quality. We propose R eliable E xpertise- A ware L anguage M odel (REALM) Fine-Tuning, a method that jointly learns the model parameters and an expertise value for each annotator. The key idea is to model each observed label as a mixture between the model’s prediction and a uniform random guess, weighted by the annotator’s expertise. An annotator with high expertise pushes the model toward their label; one with low expertise contributes little signal. Crucially, expertise is never observed: it is recovered entirely unsupervised from the structure of the training data. This mixture-based formulation was introduced in the imitation learning literature [ 3 , 2 ] , where it was shown to allow unsupervised recovery of demonstrator quality in sequential decision-making. We adapt and extend this framework to the LLM fine-tuning setting, where supervision is token-level and expertise varies across tasks in a structured multi-task setting. To our knowledge, this mixture-based approach has not previously been applied to language model training. We adapt the mixture objective to the LLM fine-tuning setting by restricting the model’s output distribution to the K answer-choice tokens of each multiple-choice task, and extend it to a multi-task setting where annotator expertise varies across tasks and is encoded in a learned matrix. We evaluate REALM on five multiple-choice question answering benchmarks [ 17 , 7 , 4 , 15 , 14 ] , fine-tuning three sizes of Flan-T5 [ 6 ] under simulated noisy annotations. Our experiments systematically vary the annotator expertise distribution, the noise type (uniform, asymmetric, and systematic), and the annotation mode (shared and exclusive), yielding a comprehensive picture of when and why expertise-aware training helps. Our main contributions are as follows: • We propose REALM, a mixture-based fine-tuning objective that jointly learns model parameters and per-annotator expertis vectors entirely unsupervised, requiring no additional supervision beyond annotator identity. • REALM consistently outperforms naive noisy SFT across datasets, model sizes, noise regimes, and in the multi-task setting where annotator expertise varies across tasks, with accuracy improvements of up to 50\% in the most adversarial setting and gains that grow with model size. • REALM remains robust when the noise type violates the model’s generative assumptions, because unreliable annotators still produce systematically worse labels regardless of the noise structure. 2 Background and Related Work 2.1 Learning from Noisy and Crowdsourced Labels Inferring ground-truth labels from multiple noisy annotators is a well-studied problem in machine learning. [ 9 ] introduced a seminal EM-based framework for estimating per-annotator error rates without access to gold labels, modeling each annotator through an individual confusion matrix. [ 23 ] extended this to a joint learning setting in which the classifier and annotator parameters are estimated simultaneously, showing consistent improvements over majority voting. [ 28 ] further incorporate item difficulty alongside annotator expertise, showing that both factors jointly determine label quality. These methods, however, are designed for fixed classification models and do not apply to the end-to-end fine-tuning of LLMs, where the learned representation and the annotator reliability must be optimised jointly through gradient descent. In NLP, annotator disagreement has received growing attention, particularly for subjective tasks such as toxicity detection and sentiment analysis [ 8 , 12 , 21 , 25 ] . [ 8 ] show that multi-task models which predict each annotator’s label separately outperform majority-vote aggregation on several binary classification tasks. [ 12 ] propose jury learning, which explicitly selects a composition of annotators at inference time to reflect a desired social perspective. While these works model disagreement as a signal to preserve, they do not address the setting where disagreement arises from differing levels of expertise on a factual task with a single correct answer, which is the setting we consider. A related line of work addresses label noise in deep learning more broadly [ 18 , 13 , 20 ] , and recent work has extended noise-robust training to the LLM fine-tuning setting [ 16 , 26 , 29 , 31 ] . These approaches typically assume a fixed noise structure, require external guidance or a clean reference set to identify noisy samples, and do not model annotator identity. In contrast, REALM requires neither: it learns annotator reliability unsupervised as part of training, with no access to ground-truth labels at any stage, and explicitly models the per-annotator source of noise rather than treating it as a property of individual examples. 2.2 Learning from Suboptimal Demonstrators Imitation learning from heterogeneous and suboptimal demonstrators is a well-studied problem in sequential decision-making. Standard behavioral cloning [ 22 ] treats all demonstrators