When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment

Robinson Vasquez Ferrer, Damla Turgut, Zhongzhou Chen, Shashank Sonkar · 2026

Read When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment by Robinson Vasquez Ferrer, Damla Turgut, Zhongzhou Chen, Shashank Sonkar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29559.txt.

Description

Ferrer and co-authors ask when a language model grader can be trusted. Rather than raise grading accuracy, they predict when the grader is likely right. High-confidence grades can then auto-process, and shaky ones go to humans. They compare three confidence routes: self-reported scores, self-consistency votes, and token probabilities. Seven models grade three education datasets. Self-reported confidence calibrated best in every condition. Voting cost five times more yet calibrated worse. Bigger models calibrated better, though gains varied by dataset.

Catalog record

CREATORRobinson Vasquez Ferrer, Damla Turgut, Zhongzhou Chen, Shashank Sonkar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28801
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.29559

Browse the full library · About the Tower of Babel