Leveraging BART to Assess CS1 C++ Programming Assignments using Rubric-based Criteria

Kelsey Rainey, Jesse Roberts · 2026

Read Leveraging BART to Assess CS1 C++ Programming Assignments using Rubric-based Criteria by Kelsey Rainey, Jesse Roberts (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03814.txt.

Introduction

Rainey and Roberts grade beginner C++ work with a rubric-aware BART model. Automated grading should mirror instructor behavior, not just chase accuracy. General-purpose language models fall short on that standard. Student submissions pair with numeric scores, letter buckets, and rubrics from many semesters of CS1. All are packed into unified sequences for a transformer.

A BART encoder-decoder with LoRA adaptation predicts scores and buckets jointly. A distribution-matching term aligns predicted and real grade spreads. That dimension is often skipped in prior work. Variants test single versus multitask training and hard versus soft labels. Rubric context plus boundary-based soft labels wins on error and distribution match. A fully tuned T5 improves distribution fidelity further. Pairwise pretraining cuts numeric error but hurts minority classes. The lesson is that rubric-guided, calibration-aware training grades more like an instructor.

Catalog record

CREATORKelsey Rainey, Jesse Roberts
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03814

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging BART to Assess CS1 C++ Programming Assignments using Rubric-based Criteria Kelsey Rainey, Jesse Roberts Abstract This paper investigates rubric-aware, multitask fine-tuning of transformer models for automated grading of introductory C++ programming assignments, with the goal of producing grade predictions that better reflect instructor grading behavior than general-purpose LLMs. Using multi-semester CS1 data, student submissions are paired with numeric scores, letter-grade buckets, and assignment rubrics, then preprocessed into unified sequences for transformer input. A BART encoder-decoder with LoRA adaptation is trained to jointly predict numeric grades and grade buckets, augmented with a distribution-matching term to align predicted and empirical grade distributions, an evaluation dimension often overlooked in prior work. Experiments compare single-task and multitask training, hard one-hot versus fuzzy and boundary-based soft labels, and rubric versus no-rubric conditions, with additional T5 and pairwise-pretrained variants. Results show that multitask BART with boundary-based soft labels and rubric context achieves lower mean absolute error and stronger grade-distribution alignment than single-task, hard-label, or code-only baselines. Fully fine-tuned T5 further improves distributional fidelity, while pairwise pretraining reduces numeric error at the cost of minority-class sensitivity. Collectively, the findings suggest that calibration-aware, rubric-guided training produces more instructor-like grading behavior than accuracy-optimized alternatives. I Introduction Automated grading tools for introductory programming courses have the potential to reduce instructor workload, provide timely feedback to students, and scale assessment to larger class sizes. However, general-purpose large language models (LLMs) present a specific problem in this context: they are typically optimized for professional or advanced programming practices rather than the conventions taught in introductory courses, particularly in courses like the CS1 course examined here which uses C++. For example, introductory C++ courses commonly require students to use using namespace std to reduce syntactic complexity while core concepts are being learned, even though standard industry practice discourages this due to potential namespace conflicts. A model that penalizes such conventions, or rewards solutions that exceed the expected scope of the assignment, may grade in ways that conflict with instructor intent and course learning objectives. This misalignment motivates the development of grading models fine-tuned on course-specific materials and rubrics. Beyond simply predicting a score, an effective automated grader should reflect the distribution of grades that an instructor would assign, distinguishing borderline submissions, avoiding collapse toward dominant grade categories, and handling the inherent subjectivity of rubric-based evaluation. These properties are not well captured by classification accuracy alone, particularly in introductory courses where grade distributions are often heavily skewed toward high marks. This paper investigates how different training configurations affect the ability of transformer models to replicate instructor grading behavior on introductory C++ assignments. Starting from a BART encoder-decoder with LoRA adaptation as a base, we systematically examine the effect of multitask learning, labeling strategy, and rubric context on grading performance. We then extend the investigation to a T5-based model and a pairwise pretraining approach to assess whether these alternatives offer meaningful improvements. Model behavior is evaluated not only through standard metrics such as MAE, accuracy, and macro-F1, but also through Jensen-Shannon divergence between predicted and true grade distributions, a measure of whether a model reproduces realistic grading behavior rather than simply maximizing point accuracy. Specifically, experiments address whether multitask learning outperforms single-task alternatives, whether soft labeling better captures grading uncertainty than hard one-hot encoding, and whether rubric context improves calibration. This work is intended as a foundation for a broader course-specific grading and feedback system, with the longer-term goal of supporting students who cannot immediately access instructional staff and need feedback aligned with their course’s specific expectations rather than general programming conventions. II Background and Related Work Artificial intelligence usage by educators and students alike has become increasingly prominent because of its ability to streamline tasks such as grading, generating feedback, explaining answers, or providing guided assistance. In introductory computer science courses, however, the impact of generative AI is difficult to measure because student experiences vary widely. Some novice programmers are able to effectively integrate AI into their workflow, while others struggle to learn both programming concepts and how to use AI tools effectively [ 1 ] . In a study by Pankiewicz [ 2 ] , students in an Object-Oriented Programming course were provided hints generated by ChatGPT. The hints are aimed to help novice students resolve compiler errors, runtime errors, or a failed unit test by explaining the error and expected solution. Results showed that 46% of students found the hints useful, while 19% reported that the hints were not helpful. Pankiewicz concluded that while the hints were useful, further research could be done into optimizing the hints to meet student needs and expectations. II-A AI in Code Grading Research on AI-assisted code grading has explored a variety of model architectures and evaluation approaches. One study compared BERT-based grading models against LSTM and TF-IDF approaches and found that BERT outperformed both alternatives because of its bidirectional encoding and stronger semantic understanding of code [ 3 ] . Other studies have identified additional limitations associated with using LLMs for code grading. Smaller models often produce unreliable feedback, and general-purpose LLMs can sometimes outperform code-specific models [ 4 ] . Subjective grading criteria, such as readability and coding style, may also introduce inconsistency and variance during training and evaluation [ 5 ] . Additionally, another study reported that AI graders tend to assign harsher grades than human evaluators [ 6 ] . II-B AI in Rubric-Based Grading Rubric-based grading has been studied extensively in short-answer assessment and shares similarities with code grading since multiple solutions may satisfy the same learning objectives. Rubrics help models generalize across varied responses by emphasizing the relevant criteria for grading. Prior work has shown that rubric-based scoring can improve grading accuracy and consistency for short-answer responses [ 7 ] . Studies using LLMs for rubric grading report moderate to strong agreement with human evaluators, particularly when the rubric is clearly structured and detailed [ 5 ] . Research focused specifically on computer science grading has also shown that the method used to provide rubrics to the model affects performance. Supplying the complete rubric at once produced results that aligned more closely with human grading than evaluating one criterion at a time. Overall, rubric-based approaches consistently outperformed models that did not use rubrics [ 8 ] . While prior research has explored AI-assisted grading, rubric-based evaluation, and the use of transformer models for code assessment independently, fewer studies have examined how these approaches can be combined into a specialized multitask grading system for introductory computer science education. Existing work often evaluates general-purpose LLMs or focuses solely on either numeric scoring or categorical assessment, rather than training a model to perform both tasks simultaneously. Additionally, many studies rely on prompt engineering with pretrained models instead of fine-tuning models on course-specific assignments and instructor grading practices. This research differs by developing a fine-tuned, multitask, rubric-based LLM designed specifically for introductory programming courses. The model incorporates rubric context directly into the grading process while jointly predicting numeric grades and categorical performance levels, allowing shared learning between related grading tasks. By combining multitask learning with course-specific rubric guidance, this work investigates whether a specialized grading model can achieve stronger agreement with instructor evaluations than existing generalized AI grading approaches. II-C Large Language Model This

Read the complete text