Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory

Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim · 2026

Read Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory by Junhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-00521.txt.

Description

Junhyuk Choi and colleagues test whether language-model judges work as stable measuring tools, not just as output scorers. Current validation looks mainly at observed outputs, giving little insight into judge stability. Their two-phase diagnostic framework, grounded in Item Response Theory, adopts the Graded Response Model. It formalises reliability as intrinsic consistency under prompt changes and human alignment with human quality judgements. Trials across varied language-model judges show the approach yields interpretable signals for spotting unreliability and its causes, aiding practical checks of judge reliability.

Catalog record

CREATORJunhyuk Choi, Sohhyung Park, Chanhee Cho, Hyeonchu Park, Bugeun Kim
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61963
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.00521

Browse the full library · About the Tower of Babel