CHiL(L)Grader : C alibrated H uman- i n-the- L oop Short-Answer Grading
Pranav Raikote (✉), Korbinian Randl, Ioanna Miliou, Athanasios Lakes, Panagiotis Papapetrou · 2026
Read CHiL(L)Grader : C alibrated H uman- i n-the- L oop Short-Answer Grading by Pranav Raikote (✉), Korbinian Randl, Ioanna Miliou, Athanasios Lakes, Panagiotis Papapetrou (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11957.txt.
Introduction
Raikote and co-authors introduce CHiL(L)Grader, a short-answer grading system that knows when to trust itself. Instruction-tuned models grade confidently even when wrong, so full automation is unsafe for high-stakes tests. The system calibrates confidence with temperature scaling, grades only sure cases, and routes the rest to humans. Teacher corrections feed back in as rubrics evolve. Across three data sets it auto-scores 35 to 65% of answers at expert-level quality. A wide quality gap between accepted and rejected answers confirms the routing.
Catalog record
| CREATOR | Pranav Raikote (✉), Korbinian Randl, Ioanna Miliou, Athanasios Lakes, Panagiotis Papapetrou |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52955 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11957
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
CHiL(L)Grader : C alibrated H uman- i n-the- L oop Short-Answer Grading Pranav Raikote (✉), Korbinian Randl, Ioanna Miliou, Athanasios Lakes, Panagiotis Papapetrou Abstract Scaling educational assessment with large language models requires not just accuracy, but the ability to recognize when predictions are trustworthy. Instruction - tuned models tend to be overconfident, and their reliability deteriorates as curricula evolve, making fully autonomous deployment unsafe in high - stakes settings. We introduce CHiL(L)Grader , the first automated grading framework that incorporates calibrated confidence estimation into a human - in - the - loop workflow. Using post - hoc temperature scaling, confidence - based selective prediction, and continual learning, CHiL(L)Grader automates only high - confidence predictions while routing uncertain cases to human graders, and adapts to evolving rubrics and unseen questions. Across three short answer grading datasets, CHiL(L)Grader automatically scores 35 − 65 % 35-65\% of responses at expert - level quality (QWK ≥ 0.80 \geq 0.80 ). A QWK gap of + 0.347 +0.347 between accepted and rejected predictions confirms the effectiveness of the confidence - based routing. Each correction cycle strengthens the model’s grading capability as it learns from teacher feedback. These results show that uncertainty quantification is key for reliable AI - assisted grading. 1 Introduction Figure 1: The CHiL(L)Grader loop over two iterations for similar responses to the same question. In Iteration I, the model predicts 3/5 with low confidence (40%), triggering teacher review; the corrected 4/5 grade is used for fine - tuning. In Iteration II, the updated model predicts 4/5 with high confidence (80%), enabling automatic acceptance. Education systems worldwide are rapidly transforming due to the expansion of higher education and the rise of online and blended learning environments [ 23 ] . While this growth improves access to education, it also increases pressure on instructional resources, as educators must deliver high-quality teaching, timely feedback, and fair assessment to larger and more diverse student cohorts. In response, Machine Learning (ML) has emerged as a promising means to support teaching and learning, through the adoption of ML-based tools, such as intelligent tutoring systems, adaptive feedback mechanisms, and personalized learning environments [ 9 ] . These developments reflect a broader shift toward data-driven, AI-supported education, further accelerated by advances in Large Language Models (LLMs). With strong capabilities in reasoning, summarization, and evaluation, LLMs have increased interest in applying ML to core educational applications, including automated assessment and feedback generation. Among these applications, assessment stands out as particularly impactful, since it shapes learning trajectories, determines academic progression, and directly affects student outcomes. Nonetheless, large-scale assessment faces a persistent trade-off between quality and feasibility. Human grading ensures accuracy and personalized feedback, but becomes costly and time-consuming [ 3 ] . LLMs offer a compelling alternative for Automated Short-Answer Grading (ASAG) tasks, demonstrating strong performance across various educational domains [ 1 , 17 , 21 ] . However, despite their competitive performance, their deployment in high-stakes educational settings remains constrained by two fundamental challenges: (i) LLMs exhibit systematic overconfidence in their predictions [ 12 , 15 , 33 ] . They routinely assign high confidence scores even to incorrect predictions, providing no reliable mechanism for teachers to determine when model outputs can be trusted and when human intervention is necessary. This miscalibration problem is particularly acute in educational contexts where the consequences of grading errors affect academic progression. (ii) Model performance degrades substantially under distribution shi-ft [ 26 ] , as systems encounter different question types, grading rubrics, and response patterns from those seen during initial training. This brittleness to distributional changes is especially problematic in educational settings where the curriculum evolves and instructors modify questions and grading rubrics [ 3 ] . These limitations prevent fully autonomous ASAG systems from being deployed in educational contexts. Effective use requires methods that reduce instructor effort, adapt over time, and produce calibrated, trustworthy confidence scores. Human-in-the-Loop (HiL) frameworks improve reliability by combining automated grading with human oversight. In this paper, we address the above limitations by proposing a calibrated HiL framework for ASAG that integrates three complementary mechanisms: post-hoc calibration to obtain reliable confidence estimates [ 15 ] , selective prediction to defer uncertain cases to human review, and a continual learning loop that incorporates human feedback to adapt to new grading conditions while mitigating catastrophic forgetting [ 7 , 30 ] . As illustrated in Figure 1 , the LLM predicts both a grade and an associated confidence score for each exam question response; high-confidence predictions are accepted automatically, while low-confidence cases are reviewed and corrected by human evaluators. These corrected samples are accumulated to iteratively retrain and recalibrate the model. In summary, our contributions are as follows: 1. Novelty. We introduce CHiL(L)Grader , the first HiL framework for ASAG that integrates (i) confidence calibration, (ii) confidence-based selective prediction, and (iii) principled human deferral into a unified system. It achieves expert-level grading performance (QWK \geq 0.80 ) on at least 30\% (and up to 65\% ) of student responses, while systematically deferring uncertain cases to human review. 2. Reliability. CHiL(L)Grader addresses the overconfidence challenge by introducing temperature scaling based on the Expected Calibration Error (ECE) [ 15 , 27 ] and empirically demonstrating consistent reductions across three data-sets and yielding calibrated confidence estimates suitable for selective prediction. 3. Adaptation. We demonstrate that HiL-based continual learning enables generalization under distribution shift. By leveraging human corrections as supervision, our approach maintains consistent improvements in performance across questions and rubrics that differ from the initial training conditions. 4. Effectiveness. Our experiments on three datasets spanning different domains, grading scales, and difficulty levels show that CHiL(L)Grader consistently improves the grading quality, while reducing the number of required manual corrections. 5. Reproducibility. Our code, model configurations, and evaluation scripts are publicly available on GitHub 1 1 1 https://anonymous.4open.science/r/chil-grading-96A3/README.md . 2 Related Work Early ASAG systems [ 4 , 20 ] relied on domain-specific semantic representations and rule-based concept matching to identify key ideas in student responses. Later work introduced statistical and semantic-similarity approaches, including vector-space and regression models for ASAG [ 11 , 24 ] . More recently, LLMs have demonstrated strong performance for ASAG, for example by showing that GPT-4, when prompted with structured rubrics, can produce scores comparable to teachers and outperform peer grading [ 17 ] . Recent advances show that LLMs can achieve teacher - level performance when guided by structured rubrics [ 17 ] , and fine - tuned models such as LLaMA - 2 and Mistral can generate richer grading feedback [ 1 ] . For longer responses, GPT - 3.5 - based feedback generation and targeted fine - tuning have further improved beyond simple score prediction [ 34 ] . Several works enhance factual grounding in ASAG using Retrieval Augmented Generation (RAG). For instance, Duong et al. [ 10 ] retrieve embedded lecture notes to guide GPT-3.5/4 grading, improving correlation with human evaluators, while Chu et al. [ 5 ] build a multi index knowledge base combining course materials and graded examples, yielding consistent zero shot gains. These approaches enhance accuracy and contextual grounding but largely assume fully automated grading. Human preference integration has largely focused on offline alignment. RLHF has been applied using Stack Overflow votes to fine tune