When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment
Robinson Vasquez Ferrer, Damla Turgut, Zhongzhou Chen, Shashank Sonkar · 2026
Read When Can We Trust LLM Graders? Calibrating Confidence for Automated Assessment by Robinson Vasquez Ferrer, Damla Turgut, Zhongzhou Chen, Shashank Sonkar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29559.txt.
Description
Ferrer and co-authors ask when a language model grader can be trusted. Rather than raise grading accuracy, they predict when the grader is likely right. High-confidence grades can then auto-process, and shaky ones go to humans. They compare three confidence routes: self-reported scores, self-consistency votes, and token probabilities. Seven models grade three education datasets. Self-reported confidence calibrated best in every condition. Voting cost five times more yet calibrated worse. Bigger models calibrated better, though gains varied by dataset.
Catalog record
| CREATOR | Robinson Vasquez Ferrer, Damla Turgut, Zhongzhou Chen, Shashank Sonkar |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28801 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.29559