Should Code Models Learn Pedagogically? A Preliminary Evaluation of Curriculum Learning for Real-World Software Engineering Tasks

Kyi Shin Khant, Hong Yi Lin, Patanamon Thongtanunam · 2025

Read Should Code Models Learn Pedagogically? A Preliminary Evaluation of Curriculum Learning for Real-World Software Engineering Tasks by Kyi Shin Khant, Hong Yi Lin, Patanamon Thongtanunam (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-03806.txt.

Introduction

Kyi Shin Khant, Hong Yi Lin, and Patanamon Thongtanunam ask whether code models learn better when training goes from easy to hard in a 2025 study. Such ordered training, called curriculum learning, has helped on made up code tasks. The team tested it on real software tasks, using code length and a complexity score to set the order.

They trained a known code model on two tasks, finding copied code and writing code summaries, on a standard benchmark. In contrast to earlier work, ordered training led to forgetting and to shortcut learning. Scores stalled after the first quarter of training. The authors call for broader tests with other models, orders, and tasks before drawing firm lessons.

Catalog record

CREATORKyi Shin Khant, Hong Yi Lin, Patanamon Thongtanunam
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-03806

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Should Code Models Learn Pedagogically? A Preliminary Evaluation of Curriculum Learning for Real-World Software Engineering Tasks Kyi Shin Khant, Hong Yi Lin, Patanamon Thongtanunam Abstract Learning-based techniques, especially advanced pre-trained models for code have demonstrated capabilities in code understanding and generation, solving diverse software engineering (SE) tasks. Despite the promising results, current training approaches may not fully optimize model performance, as they typically involve learning from randomly shuffled training data. Recent work shows that Curriculum Learning (CL) can improve performance on code-related tasks through incremental learning based on the difficulty of synthetic code. Yet, the effectiveness of CL with conventional difficulty measures in SE tasks remains largely unexplored. In this study, we explore two conventional code metrics: code length and cyclomatic complexity to determine the difficulty levels. We investigate how the pre-trained code model (CodeT5) learns under CL, through the tasks of code clone detection and code summarization. Our empirical study on the CodeXGLUE benchmark showed contrasting results to prior studies, where the model exhibited signs of catastrophic forgetting and shortcut learning. Surprisingly, model performance saturates after only the first quartile of training, potentially indicating a limit in the model’s representation capacity and/or the task’s inherent difficulty. Future work should further explore various CL strategies with different code models across a wider range of SE tasks for a more holistic understanding. I Introduction Learning-based techniques, especially advanced pre-trained models for code have shown potential in code understanding and generation. These capabilities have proven to be useful for a diverse range of software engineering (SE) tasks, such as vulnerability detection [ 1 ] , bug-fixing [ 2 ] , and code translation [ 3 ] . Whilst these results are promising, code models often still struggle with longer [ 4 ] and more complex programs [ 5 ] . In recent years, a plethora of research has investigated code-oriented pre-training [ 6 ] as the primary method for improving code models, however, the effects of employing different fine-tuning strategies for downstream SE tasks still lacks attention. By comparison, curriculum learning (CL), a difficulty-based training strategy inspired by pedagogical learning principles [ 7 ] has demonstrated promising results in domains such as natural language processing and computer vision [ 8 ] . For code understanding, CL has shown potential in synthetic programming challenges involving code completion [ 9 ] and code execution [ 10 ] . For real-world SE, CL has been effective in tasks such as code clone detection [ 11 ] and automated program repair [ 12 ] . However, these studies rely on semantic preserving transformations to create difficult code, rather than natural code difficulty. They also do not examine how the model is gaining competency at each stage of CL. In this study, we set out to investigate whether CL with conventional difficulty measures can enhance the understanding and generative capabilities of pre-trained code models with real-world software. We also investigate the stages of this learning process, and how each difficulty affects the model. We experimented with two well-established SE tasks, i.e., code clone detection and code summarization [ 6 ] . We structured the CL training schedules by organizing the fine-tuning data into four difficulty levels (easy, medium, hard, and very hard) based on conventional difficulty metrics i.e., length ( L ) and cyclomatic complexity ( C ). We explore both sequential ( s ) and reverse sequential ( r ) CL schedules, i.e., fine-tuning the model from easy to hard, and vice versa. Through an empirical study of CodeT5 [ 13 ] , using the CodeXGLUE benchmark [ 3 ] , we address the following two research questions: RQ1. Can curriculum learning with conventional difficulty metrics enhance the performance of code models in code clone detection and code summarization ? Findings: In our experiments with CodeT5, CL with conventional difficulty measures did not outperform the traditional random schedule in the two selected tasks. RQ2. How does each difficulty level impact model performance during CL? Findings: Interestingly, model performances reach saturation with only 25% of the training data. In other words, after initially fine-tuning on either the easy or very hard subset, additional data from the subsequent difficulty levels yield only minimal performance gains. Our RQ1 findings suggest that, despite promising results of CL with difficult synthetic code, scheduled training based on conventional difficulty of code had little impact on CodeT5’s capabilities in the selected tasks. Instead, RQ2 reveals that CodeT5 can sufficiently learn common patterns for these tasks from limited data, and additional training on other difficulty levels only marginally enhances these capabilities. Nonetheless, this may be due to the model’s limited representation capacity and/or the natural difficulty of the task. For a more comprehensive evaluation, future studies should include more diverse code models, datasets and difficulty measures. Contribution. To the best of our knowledge, we are the first to investigate the learning process of code models during CL with conventional difficulty metrics. Our study unveils an interesting phenomenon, providing motivation for further research into how code models learn downstream SE tasks. II Background and Related Work II-A Neural Code Models The success of neural language models in the natural language domain has inspired researchers to apply similar methods to programming languages. To alleviate the software engineering workload, a large plethora of research has explored automating SE tasks, such as vulnerability detection [ 1 ] , bug-fixing [ 2 ] , code translation [ 3 ] , code clone detection [ 14 ] and code summarization [ 15 ] . Specifically, the techniques involve training neural language models on data from public repositories such as GitHub and Stack Overflow [ 13 ] . Leveraging the naturalness properties of human written code, these neural code models have learned useful patterns that unlock their potential as practical software development assistants [ 16 ] . II-B Curriculum Learning Replicating the pedagogical learning process of humans, CL is a training strategy that gradually increases the difficulty of examples presented to the model [ 7 ] , which contrasts with the conventional unordered training schedule. CL has been effective for a wide range of model architectures and tasks [ 17 , 8 ] . It improves model performance on complex examples by aligning training schedules with the model’s skill-acquiring pace [ 18 ] . Using generated programs, Naïr et al. [ 9 ] investigated the effects of CL in code completion, whilst Liu et al. [ 10 ] focused on the task of code execution in competitive programming. Although conventional difficulty measures such as length and cyclomatic complexity were investigated, these studies focused on synthetic programming problems, which may not generalise to real-world software. In terms of real-world SE, CL has been employed in tasks, such as code clone detection, code search [ 11 ] and automated program repair [ 12 ] , however, the methods largely focused on augmenting code to create difficult synthetic examples. Additionally, past studies mainly focused on the final result, rather than investigating how the code model improves during CL, leaving us with an opaque understanding of the process. Past studies have also yet to investigate the code-to-text task of code summarization. III Study Design III-A Overview To investigate whether CL can improve models’ code understanding,

Read the complete text