SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models

Peichao Lai, Kexuan Zhang, Yi Lin, Linyihan Zhang, Feiyang Ye, Jinhao Yan, Yanwei Xu, Conghui He, Yilei Wang, Wentao Zhang, Bin Cui · 2025

Read SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models by Peichao Lai, Kexuan Zhang, Yi Lin, Linyihan Zhang, Feiyang Ye, Jinhao Yan, Yanwei Xu, Conghui He, Yilei Wang, Wentao Zhang, Bin Cui (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07247.txt.

Description

Peichao Lai and co-authors build a benchmark for grading short answers in a 2025 paper. Short answer scoring matters in education and testing, but past tools give coarse scores with little reasoning. Large language models may grade with no training examples, yet they can be biased and hard to audit. A finer test set is needed.

SAS-Bench gives step-wise scores, expert error labels, and varied question types from real subject exams. The open data set holds over a thousand questions and more than four thousand student answers, each marked by experts. Tests across models show science questions are hardest, and a few examples in the prompt help a lot.

Catalog record

CREATORPeichao Lai, Kexuan Zhang, Yi Lin, Linyihan Zhang, Feiyang Ye, Jinhao Yan, Yanwei Xu, Conghui He, Yilei Wang, Wentao Zhang, Bin Cui
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-07247

Browse the full library · About the Tower of Babel