Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specific Adaptation

Abigail Victoria Gurin Schleifer Moriah Ariely Beata Beigman Klebanov, Asaf Salman Giora Alexandron · 2026

Read Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specific Adaptation by Abigail Victoria Gurin Schleifer Moriah Ariely Beata Beigman Klebanov, Asaf Salman Giora Alexandron (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07647.txt.

Description

Abigail Victoria Gurin Schleifer and four co-authors examine automated scoring of short student answers, now often done by large language models with a few examples rather than fine-tuned graders. The risk sits in the middle of the quality range: partly correct answers need nuanced judgment, and errors there could unfairly penalize students whose understanding is still developing.

They compared three few-shot language models, a fine-tuned encoder, and a human expert on hundreds of biology answers. All AI systems degraded sharply on mid-range responses, while handling clear right and wrong answers well. The degradation shrank as task-specific adaptation increased, with the fine-tuned encoder best. Human agreement stayed highest and stable across the whole quality spectrum.

Catalog record

CREATORAbigail Victoria Gurin Schleifer Moriah Ariely Beata Beigman Klebanov, Asaf Salman Giora Alexandron
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.07647

Browse the full library · About the Tower of Babel