Quantifying Label-Induced Bias in Large Language Model Self- and Cross-Evaluations
Muskan Saraf, Sajjad Rezvani Boroujeni, Justin Beaudry, Hossein Abedi, Tom Bush · 2025
Read Quantifying Label-Induced Bias in Large Language Model Self- and Cross-Evaluations by Muskan Saraf, Sajjad Rezvani Boroujeni, Justin Beaudry, Hossein Abedi, Tom Bush (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-21164.txt.
Description
Five researchers show that who a text is claimed to be written by can swing how language models grade it, in a 2025 paper. Models now judge text quality widely, but the fairness of those judgments is under-tested. The team had three leading models each write blog posts, then had all three grade every post under four labels: none, the true author, and two false authors. Scores were normalized for comparison.
One model's name acted like a halo, lifting scores no matter who wrote the text, while another's name dragged scores down. False labels flipped preference rankings by up to fifty points. One model harshly downgraded its own work when honestly labeled, while another boosted its own. The authors warn that model-as-judge setups need blind protocols and multi-model checks to stay fair.
Catalog record
| CREATOR | Muskan Saraf, Sajjad Rezvani Boroujeni, Justin Beaudry, Hossein Abedi, Tom Bush |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-21164