Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives
Yinuo Xu, Veronica Derricks, Allison Earl, David Jurgens · 2025
Read Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectives by Yinuo Xu, Veronica Derricks, Allison Earl, David Jurgens (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02853.txt.
Description
Yinuo Xu, Veronica Derricks, Allison Earl, and David Jurgens tackle disagreement among human labelers in a 2025 computing paper. When people label text for tasks like judging tone, they often disagree in patterned ways linked to who they are. Models that force one right answer can erase those real differences.
Their model sends each input to expert subnetworks chosen by the labeler's background traits, so group-level views stay distinct. It did well across groups, above all on sets with high disagreement. The team also tested fake labels made by a language model playing set roles. Those labels matched human ones fairly well, and mixing fake with real data helped when coverage was sparse. The best mix depended on the shape of each dataset.
Catalog record
| CREATOR | Yinuo Xu, Veronica Derricks, Allison Earl, David Jurgens |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-02853