Evaluating how LLM annotations represent diverse views on contentious topics

Megan A. Brown, Shubham Atreja, Libby Hemphill, Patrick Y. Wu · 2025

Read Evaluating how LLM annotations represent diverse views on contentious topics by Megan A. Brown, Shubham Atreja, Libby Hemphill, Patrick Y. Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-23243.txt.

Description

When AI labels touchy content, whose view does it take? A 2025 paper by Megan A. Brown and co-authors checks whether language models lean toward majority groups when they annotate disputed material. Such labeling is now sold as a cheap stand-in for human coders in research and industry.

Across four tasks and four data sets, the models did not systematically side against any demographic group of human annotators. Instead, different models tended to lean the same way on the same groups within a data set. How much the human coders disagreed with each other, a sign of a hard item, predicted model agreement far better. The results calm one fear while flagging shared leanings across models.

Catalog record

CREATORMegan A. Brown, Shubham Atreja, Libby Hemphill, Patrick Y. Wu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-23243

Browse the full library · About the Tower of Babel