Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustness
Haotian Deng, Chris Farber, Jiyoon Lee, David Tang · 2026
Read Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustness by Haotian Deng, Chris Farber, Jiyoon Lee, David Tang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-08843.txt.
Introduction
Haotian Deng and three co-authors test chat models as graders of short student replies. A teacher gives a rubric, a list that says what earns full or part marks. Student words vary a lot, so part marks are hard to set fair. Chat models look like a cheap fix, but their marks must match the rubric, not their own taste.
The team runs a broad check of model graders on rubric tasks. They look at how close marks sit to human marks, how sure the model is, and how marks hold up when replies are tweaked. A reader who runs a class or a test firm gets facts on trust. The work shows where a model grader can help, and where a human must still check.
Catalog record
| CREATOR | Haotian Deng, Chris Farber, Jiyoon Lee, David Tang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-08843
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rubric-Conditioned LLM Grading: Alignment, Uncertainty, and Robustness Haotian Deng, Chris Farber, Jiyoon Lee, David Tang Abstract Automated short-answer grading (ASAG) remains a challenging task due to the linguistic variability of student responses and the need for nuanced, rubric-aligned partial credit. While Large Language Models (LLMs) offer a promising solution, their reliability as automated judges in rubric-based settings requires rigorous assessment. In this paper, we systematically evaluate the performance of LLM-judges for rubric-based short-answer grading. We investigate three key aspects: the alignment of LLM grading with expert judgment across varying rubric complexities, the trade-off between uncertainty and accuracy facilitated by a consensus-based deferral mechanism, and the model’s robustness under random input perturbations and adversarial attacks. Using the SciEntsBank benchmark and Qwen 2.5-72B, we find that alignment is strong for binary tasks but degrades with increased rubric granularity. Our “Trust Curve” analysis demonstrates a clear trade-off where filtering low-confidence predictions improves accuracy on the remaining subset. Additionally, robustness experiments reveal that while the model is resilient to prompt injection, it is sensitive to synonym substitutions. Our work provides critical insights into the capabilities and limitations of rubric-conditioned LLM judges, highlighting the importance of uncertainty estimation and robustness testing for reliable deployment. 1 Introduction Short-answer grading shows up everywhere –- university courses, online education, and compliance or safety checks –- yet it remains hard to automate. Free-form responses vary in phrasing and partial correctness; graders care about specific rubric criteria (key ideas, justification, units, reasoning steps), not just surface similarity. Hand-crafted rules rarely cover the long tail of valid answers, and fully manual grading does not scale. At the same time, collecting large, task-specific labeled datasets is expensive and often impossible. Recent “LLM-as-a-judge” approaches are promising, but out of the box, they can be unreliable, inconsistent, or brittle to harmless rewordings, especially when we only have the rubric and a small number of graded examples. Research Question: We study the rubric-only (or low-label) grading setting: Given a textual rubric and many free-form responses, to what extent can we automatically assign reliable scores with minimal human labeling? We decompose this into three specific research questions (RQs): • RQ1 (Alignment): To what extent does the model align with expert judgments on rubric criteria, and how does this alignment vary with rubric complexity? • RQ2 (Uncertainty): Can we improve system reliability by identifying uncertain predictions and selectively withdrawing them? • RQ3 (Robustness): How stable is the grader under paraphrases, minor perturbations, and adversarial attacks? This setting is relatively underexplored in current evaluation work. We investigate a systematic and pragmatic approach that treats the rubric as the primary supervision signal, uses a small number of graded anchors for light calibration, and includes an explicit option to defer to a human when confidence is low. Our goal is not to replace human graders but to understand how far rubric-conditioned judging can go under realistic data constraints. 2 Related Work Automated Short Answer Grading (ASAG). Traditional ASAG methods relied on feature engineering or fine-tuning BERT-based models, which required collecting hundreds of labeled examples for every new question to achieve high accuracy ( 1 ; 3 ) . LLM-as-a-judge (LAJ) recently emerged as a line of research in natural language processing and machine learning ( 6 ; 10 ) . This paradigm uses LLMs to evaluate outputs by following specific instructions or criteria, eliminating the need for large labeled training datasets ( 6 ) . 4 demonstrate the utility of LLMs like GPT-4 for enhancing assessment in parallel functional programming courses, focusing on improving grading efficiency and objectivity. 5 similarly explores the potential of LLMs for grading in medical education. Others have explored integrating LLMs into human-in-the-loop workflows to refine feedback ( 2 ; 20 ) or utilizing Retrieval-Augmented Generation (RAG) to enhance grading accuracy ( 15 ) . While these works focus on domain-specific applications and efficiency with human feedback, our work addresses the safety and robustness required for autonomous deployment. We systematically test adversarial resilience against prompt injections and introduce a consensus-based deferral mechanism to actively mitigate hallucination risks, pushing the boundary of reliable autograding without direct human intervention. 2.1 Improving LLM Judges Several lines of work aim to increase accuracy, robustness, and stability of LAJ by intervening at three layers – how the model is prompted, how it is trained, and how outputs are aggregated. Prompting. Chain-of-Thought (CoT) prompting elicits explicit intermediate reasoning, often improving robustness and transparency on reasoning-heavy tasks such as mathematics relative to standard short-answer prompting ( 19 ) . Notably, automatic chain-of-thought achieves comparable performance to manual CoT. ( 22 ) Pairwise evaluation is shown to have significant improvement over direct scoring and is more in agreement with human judgment. ( 12 ) Training. 8 investigate fine-tuning LAJ via reinforcement learning on reasoning traces -— using both offline and online RL -– to strengthen consistency and deliberative quality in judge outputs ( 8 ) . Aggregation. Multi-run or multi-model aggregation can reduce variance by averaging or reconciling diverse signals. While not a judge per se, MAATS demonstrates the value of multi-LLM feedback loops in translation, combining multiple agents and MQM-style evaluation to improve downstream quality ( 17 ) . Relatedly, broader LLM evaluation work has explored experimental designs that aggregate outputs across samples to stabilize estimates, a pattern that can inform LAJ evaluation setup ( 9 ; 13 ) . Our consensus mechanism draws inspiration from Self-Consistency ( 18 ) , which demonstrates that aggregating multiple sampling paths improves reasoning performance. We extend this by using agreement not just to improve the answer, but as a proxy for confidence to trigger human deferral. 2.2 Evaluating LLM Judges Correlation to Human Judgment. A central question for LAJ is how closely model judgments align with expert labels. 7 conduct a large-scale cross-evaluation of 54 LLM judges on expert-labeled datasets and report both correlation-based agreement and Cohen’s \kappa as primary criteria ( 7 ) . 16 study multilingual, multi-topic fact-checking and find that LLM judges show limited accuracy and a high no-response rate, especially outside high-resource languages and familiar topics ( 16 ) . Adversarial Robustness. Recent studies also probe the robustness of LAJ under adversarial and safety-critical conditions. 23 show that “null models” producing constant outputs can secure surprisingly high win rates on some automatic judge benchmarks, revealing brittleness in pairwise-comparison-style evaluation pipelines ( 23 ) . 11 propose goal-oriented prompt attacks that elicit harmful behaviors with high success rates, indicating that judge prompts and scoring rules can be exploited ( 11 ) . Complementing these findings, 21 introduce JADE, a linguistics-based safety evaluation platform, and demonstrate that simple linguistic perturbations (including random mutations) can circumvent safety guardrails and induce responses to dangerous prompts ( 21 ) . Collectively, these results argue for robustness-aware judge design and for adversarially resilient evaluation protocols. 3 Methods 3.1 Core Approach We developed a rubric-conditioned grading pipeline using Large Language Models to evaluate student short answers. Unlike traditional supervised methods that require fine-tuning, our approach treats the rubric as the primary supervision signal. To systematically evaluate reliability, we employ a multi-faceted evaluation framework corresponding to our three research questions. First, we assess alignment by comparing model grades against expert labels across varying rubric complexities (2-way, 3-way, and 5-way). Second,