LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories
Krishnapriya Vishnubhotla, Sowmya Vajjala, Akriti Vij, Isar Nejadgholi · 2026
Read LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categories by Krishnapriya Vishnubhotla, Sowmya Vajjala, Akriti Vij, Isar Nejadgholi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31381.txt.
Description
Vishnubhotla and colleagues test how evenly machine judges evaluate safety in a reference-free setup. Large language models prove shaky judges for safety issues in machine-made advice in watched domains like finance, though they do better on overt harmful content like violence. Wobble varies a lot by safety criterion and shifts with the language and wording of the content. Judges also differ sharply on the same output across domains, rules and languages. The findings urge caution in using language models as evaluators, with practical tips for users.
Catalog record
| CREATOR | Krishnapriya Vishnubhotla, Sowmya Vajjala, Akriti Vij, Isar Nejadgholi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58127 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.31381