Skewed score: a statistical framework to assess autograders
Magda DuboisHarry CoppockMario GiulianelliTimo Flesch, Lennart Luettgau, Cozmin Ududec UK AI Security InstituteLondon, United Kingdom · 2025
Read Skewed score: a statistical framework to assess autograders by Magda DuboisHarry CoppockMario GiulianelliTimo Flesch, Lennart Luettgau, Cozmin Ududec UK AI Security InstituteLondon, United Kingdom (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03772.txt.
Description
Magda Dubois and co-authors build a statistical check for AI graders in a 2025 paper. Language models now grade the output of other language models at scale. Such autograders save labor, yet they show mixed reliability and hidden biases of many kinds.
Their framework rests on Bayesian generalized linear models of the grading outcome. Scores are modeled as a joint function of grader traits and item traits, such as answer length. That split lets bias show itself apart from true quality gaps. Teams can audit a grader and answer their main question in one model. Fairer automated grading is vital as these judges spread through research and hiring.
Catalog record
| CREATOR | Magda DuboisHarry CoppockMario GiulianelliTimo Flesch, Lennart Luettgau, Cozmin Ududec UK AI Security InstituteLondon, United Kingdom |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-03772