PyMETA: A Benchmark Dataset for Hierarchical Student Code Error Classification with Python-Interpreter-Based Labels
Chuyue Li, Ziqi Tang, Jingyi Wang, Yu Wu, Kazuma Hashimoto, Lingyu Gao · 2026
Read PyMETA: A Benchmark Dataset for Hierarchical Student Code Error Classification with Python-Interpreter-Based Labels by Chuyue Li, Ziqi Tang, Jingyi Wang, Yu Wu, Kazuma Hashimoto, Lingyu Gao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30610.txt.
Introduction
Chuyue Li and five colleagues release PyMETA, a benchmark for classifying errors in student Python code, released in 2026 for education research. Teachers and tutoring tools need to name what went wrong in a submission, not just mark it wrong. Existing datasets are small, lack multi-error analysis and use fragmented error labels. PyMETA instead labels errors with the Python interpreter's own exception hierarchy.
The dataset holds 48,646 student submissions. Every sample carries a single-error label, and a diagnostic subset of ninety-seven samples was annotated by experts for multiple errors. Its taxonomy has three levels, from a binary error split down to fourteen fine-grained types. The team tested two finetuned models and four prompted language models. Prompted models still trailed the finetuned smaller ones. Most models over-used the Logic Error label, and the best multi-error score came from Gemini 2.5 Pro.
Catalog record
| CREATOR | Chuyue Li, Ziqi Tang, Jingyi Wang, Yu Wu, Kazuma Hashimoto, Lingyu Gao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30610
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PyMETA: A Benchmark Dataset for Hierarchical Student Code Error Classification with Python-Interpreter-Based Labels Chuyue Li, Ziqi Tang, Jingyi Wang, Yu Wu, Kazuma Hashimoto, Lingyu Gao Abstract With the advancement of Large Language Models (LLMs), code error detection has extended beyond traditional IDE diagnostics to context-sensitive debugging in educational scenarios. However, existing approaches lack large-scale datasets, multi-error analysis, and unified error taxonomies. To address this, we introduce PyMETA , a large-scale Python code error classification dataset of 48,646 student submissions, with single-error labels for all samples and a diagnostic subset of 97 expert-annotated multi-error samples. The dataset uses a three-level hierarchical taxonomy, from a binary error/no-error split down to 14 fine-grained error types grounded in Python’s official exception hierarchy. We evaluate multi-level classification tasks on two finetuned models and four LLMs with prompting, comparing their classification performance and runtime cost. For multi-error prompting, the best model, Gemini 2.5 Pro, achieves 81.8% macro F1 under the contains criterion. We observe that: 1) prompted LLMs still underperform finetuned smaller models; 2) models exhibit significant disparities across error types; 3) most LLMs over-classify code as Logic Error , with GPT-3.5 showing the highest Logic Error Overprediction Rate and Gemini 2.5 Pro the lowest. Our work establishes a data foundation and provides insights for LLM-based code error research. 1 Introduction As Large Language Models (LLMs) grow more capable, code-related applications span programming education 15 ; 20 ; 1 , IDE/compiler integration 3 ; 10 ; 18 , and engineering 5 . These settings motivate a range of code-diagnosis tasks, including multi-error classification 15 ; 1 , error localization 8 , and error correction 8 . However, this area still faces several challenges. First, public code error datasets lack broad coverage of problem types and error categories, large-scale samples, and multi-error analysis 8 ; 1 . Second, existing error categorizations are often fragmented or ad hoc, lacking a unified taxonomy bridging human understanding and IDE diagnostics 15 ; 8 . Third, standardized benchmarks for evaluating LLMs’ code error detection are scarce, particularly under training-free or low-resource settings 16 . To address these, this paper makes the following contributions: 1) We introduce PyMETA , a large-scale Python code error dataset of 48,646 student submissions across 155 problems 1 1 1 See Section 5.1 for the finetuning split. A few problems are excluded during pre-processing. from 579 users. All samples are assigned single-error labels derived from IDE execution signals, while a 97-sample diagnostic subset is further annotated by experts for multiple concurrent errors. This subset enables us to examine whether apparent single-error failures involve additional underlying errors. 2 2 2 The dataset, splits, annotation guidelines, and evaluation scripts are available at https://github.com/Circle-Cat/pymeta . 2) We establish a multi-level benchmark evaluating representative LLMs under both finetuning and prompting, revealing model-specific biases and performance gaps for future LLM-based code diagnosis research. 3) Our analyses reveal: (i) prompted LLMs substantially underperform finetuned smaller models; (ii) models exhibit a strong bias toward over-predicting Logic Error (single-error \mathrm{LER}_{\mathrm{over}} 17.6–92.8%; multi-error 55.6–75.6%); (iii) multi-error prompting remains challenging even for strong models (best macro F1 81.8% under the contains criterion); and (iv) entropy and confusion analyses expose model uncertainty in complex multi-error cases. We evaluate two finetuned models (CodeBERT, CodeLlama-7B) and four prompted LLMs (DeepSeek-V3, Gemini 2.5 Pro, GPT-3.5, GPT-4o). Our main finding is that finetuning a small model still works better than prompting much larger ones on this task: CodeLlama-7B reaches 80.6% macro F1 on single-error classification, while the best prompted model, Gemini 2.5 Pro, reaches 71.9%. Prompted models also share a common weakness: they over-predict Logic Error and do poorly on rare error types, showing where future work on LLM-based code diagnosis is most needed. We also report inference cost, which differs by more than ten times across models. 2 Related Work 2.1 Dataset for Code Error Classification Code error detection, classification, localization, and analysis have long been studied in programming education, and the growing use of data-driven and LLM-based methods has increased the demand for high-quality large-scale datasets 9 . Some studies 10 ; 2 ; 1 rely on existing open-source datasets like the AOJ ITP1 dataset. 3 3 3 https://onlinejudge.u-aizu.ac.jp/courses/lesson/2/ITP1 (accessed on 2025-12-30). Others 8 ; 19 introduce course based C language datasets such as COJ2022 and CPE28. Taking AOJ ITP1 as an example, it is based on 44 problems and thus does not capture diverse problems and error cases. These course-based datasets often exhibit limited problem diversity, homogeneous user populations, and small sample sizes, and typically lack fine-grained multi-error annotations. To tackle these limitations and to contribute an open dataset for code error classification, we introduce PyMETA : a Python code error classification dataset. 2.2 Code Error Detection with Large Models Recent studies aim to improve code error classification by training strategies and model architectures, including combining a CodeT5 encoder with ML-KNN 2 , finetuning two BERT variants for multi-label classification 1 , and introducing a model that integrates BiLSTM and TextCNN 19 . Recent work also suggests that Code LLMs struggle to reason about runtime behavior 17 ; 7 . However, we hypothesize that current LLMs may already possess inherent code error classification capabilities, yet systematic analyses remain limited. We thus compare common LLMs on single- and multi-error classification using basic finetuning and training-free approaches, analyzing potential biases. All PyMETA labels derive from OJ execution output, including Logic Error for code that executes but fails test cases. This provides an execution-grounded standard, different from current models frequently misalign (Appendix A.11.4 ). 4 4 4 Two patterns emerge: predicting Logic Error for parse-time failures, and Syntax Error for executable code. 3 Dataset Construction In this section, we will introduce our proposed dataset. The data sources and data processing are detailed in Appendix A.4 and A.5 . 3.1 Platform and Data Source 3.2 Dataset Overview feature definition User ID unique identifier of the student Name problem type Question ID unique identifier of the problem Question provides the problem description Answer sample of the correct code solution Attempt ID attempt number StudentAnswer contains the student’s submitted code TestOutcome the Online Judge error messages outputs AttemptStepID denotes the step identifier for the attempt Table 1: Dataset Feature Descriptions. Column names and definitions for the 9 features included in each PYMETA sample. We constructed a Python code error dataset PyMETA from student submission logs collected through the Circle Cat online learning platform (see Appendix A.1 ). The dataset spans 22 problem types across 155 distinct problems, with 48,646 real code submissions from 579 users and an expert-annotated multi-error diagnostic subset of 97 samples. Each sample contains a problem description, the student’s Python code submission, its compilation and execution results from an online judge platform, Moodle CodeRunner, 5 5 5 CodeRunner: a Moodle question type for programming courses. http://coderunner.org.nz (accessed on 2025-12-30) and the correct code solution. Each sample is described by 9 features (Table 1 ), and submissions are categorized into 18 error types. The distributions of problem IDs and user IDs are shown in Figure 7 and Figure 7 , respectively. In particular, the distribution of problem names and IDs is shown in Figure 7 , and the distribution of user IDs is presented in Figure 7 . In addition, we provide an