SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models
Peichao Lai, Kexuan Zhang, Yi Lin, Linyihan Zhang, Feiyang Ye, Jinhao Yan, Yanwei Xu, Conghui He, Yilei Wang, Wentao Zhang, Bin Cui · 2025
Read SAS-Bench: A Fine-Grained Benchmark for Evaluating Short Answer Scoring with Large Language Models by Peichao Lai, Kexuan Zhang, Yi Lin, Linyihan Zhang, Feiyang Ye, Jinhao Yan, Yanwei Xu, Conghui He, Yilei Wang, Wentao Zhang, Bin Cui (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07247.txt.
Description
Peichao Lai and co-authors build a benchmark for grading short answers in a 2025 paper. Short answer scoring matters in education and testing, but past tools give coarse scores with little reasoning. Large language models may grade with no training examples, yet they can be biased and hard to audit. A finer test set is needed.
SAS-Bench gives step-wise scores, expert error labels, and varied question types from real subject exams. The open data set holds over a thousand questions and more than four thousand student answers, each marked by experts. Tests across models show science questions are hardest, and a few examples in the prompt help a lot.
Catalog record
| CREATOR | Peichao Lai, Kexuan Zhang, Yi Lin, Linyihan Zhang, Feiyang Ye, Jinhao Yan, Yanwei Xu, Conghui He, Yilei Wang, Wentao Zhang, Bin Cui |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-07247