Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim · 2026
Read Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory by Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-00521.txt.
Description
Junhyuk Choi and colleagues test whether language-model judges work as stable measuring tools, not just as output scorers. Current validation looks mainly at observed outputs, giving little insight into judge stability. Their two-phase diagnostic framework, grounded in Item Response Theory, adopts the Graded Response Model. It formalises reliability as intrinsic consistency under prompt changes and human alignment with human quality judgements. Trials across varied language-model judges show the approach yields interpretable signals for spotting unreliability and its causes, aiding practical checks of judge reliability.
Catalog record
| CREATOR | Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61963 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.00521