Evaluating how LLM annotations represent diverse views on contentious topics
Megan A. Brown, Shubham Atreja, Libby Hemphill, Patrick Y. Wu · 2025
Read Evaluating how LLM annotations represent diverse views on contentious topics by Megan A. Brown, Shubham Atreja, Libby Hemphill, Patrick Y. Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23243.txt.
Description
When AI labels touchy content, whose view does it take? A 2025 paper by Megan A. Brown and co-authors checks whether language models lean toward majority groups when they annotate disputed material. Such labeling is now sold as a cheap stand-in for human coders in research and industry.
Across four tasks and four data sets, the models did not systematically side against any demographic group of human annotators. Instead, different models tended to lean the same way on the same groups within a data set. How much the human coders disagreed with each other, a sign of a hard item, predicted model agreement far better. The results calm one fear while flagging shared leanings across models.
Catalog record
| CREATOR | Megan A. Brown, Shubham Atreja, Libby Hemphill, Patrick Y. Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-23243