Quantifying and Predicting Disagreement in Graded Human Ratings

Leixin Zhang Çağrı Çöltekin · 2026

Read Quantifying and Predicting Disagreement in Graded Human Ratings by Leixin Zhang Çağrı Çöltekin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01168.txt.

Description

Leixin Zhang and one co-author study disagreement in graded human ratings of inappropriate language, covering offensive language, hate speech, and toxicity perception. Annotators do not always agree, and such splits are now seen as inherent to many tasks. Items differ in how much divergence they elicit. The authors ask whether the degree of split can be predicted from textual features.

They propose an Opposition Index that quantifies perspective opposition among annotators on an item. They study prediction of both variance and high-opposition cases. Estimated and observed variance correlate moderately, and two prediction approaches perform comparably. Items with high opposition are harder to predict and are often underestimated by models, a warning for systems that treat labels as settled.

Catalog record

CREATORLeixin Zhang Çağrı Çöltekin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.01168

Browse the full library · About the Tower of Babel