Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives

Yinuo Xu, Veronica Derricks, Allison Earl, David Jurgens · 2025

Read Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives by Yinuo Xu, Veronica Derricks, Allison Earl, David Jurgens (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02853.txt.

Description

Yinuo Xu, Veronica Derricks, Allison Earl, and David Jurgens tackle disagreement among human labelers in a 2025 computing paper. When people label text for tasks like judging tone, they often disagree in patterned ways linked to who they are. Models that force one right answer can erase those real differences.

Their model sends each input to expert subnetworks chosen by the labeler's background traits, so group-level views stay distinct. It did well across groups, above all on sets with high disagreement. The team also tested fake labels made by a language model playing set roles. Those labels matched human ones fairly well, and mixing fake with real data helped when coverage was sparse. The best mix depended on the shape of each dataset.

Catalog record

CREATORYinuo Xu, Veronica Derricks, Allison Earl, David Jurgens
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE83 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-02853

Browse the full library · About the Tower of Babel