Skewed score: a statistical framework to assess autograders

Magda DuboisHarry CoppockMario GiulianelliTimo Flesch, Lennart Luettgau, Cozmin Ududec UK AI Security InstituteLondon, United Kingdom · 2025

Read Skewed score: a statistical framework to assess autograders by Magda DuboisHarry CoppockMario GiulianelliTimo Flesch, Lennart Luettgau, Cozmin Ududec UK AI Security InstituteLondon, United Kingdom (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03772.txt.

Description

Magda Dubois and co-authors build a statistical check for AI graders in a 2025 paper. Language models now grade the output of other language models at scale. Such autograders save labor, yet they show mixed reliability and hidden biases of many kinds.

Their framework rests on Bayesian generalized linear models of the grading outcome. Scores are modeled as a joint function of grader traits and item traits, such as answer length. That split lets bias show itself apart from true quality gaps. Teams can audit a grader and answer their main question in one model. Fairer automated grading is vital as these judges spread through research and hiring.

Catalog record

CREATORMagda DuboisHarry CoppockMario GiulianelliTimo Flesch, Lennart Luettgau, Cozmin Ududec UK AI Security InstituteLondon, United Kingdom
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-03772

Browse the full library · About the Tower of Babel