Self-Evolving Curriculum for LLM Reasoning

Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, Ehsan Kamalloo · 2025

Read Self-Evolving Curriculum for LLM Reasoning by Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, Ehsan Kamalloo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14970.txt.

Introduction

Xiaoyin Chen and co-authors (2025) present a method letting a language model choose its practice order while learning to reason. In reinforcement learning, the sequence of training problems shapes how well the model ends. Random order is a weak default, and hand-made orders lean on guesswork. The authors' method, called Self-Evolving Curriculum, treats each problem category as an arm of a bandit machine. A category might be a difficulty level or a problem type.

At each step, the method picks the category with the best learning gain, read from the size of the policy update signal. The picker learns by a simple temporal-difference update. Tests covered planning, inductive reasoning, and mathematics. Models trained this way reasoned better and handled harder unseen problems. Training on several domains at once stayed more balanced. The authors offer the method as a cheap, automatic upgrade for reasoning training.

Catalog record

CREATORXiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, Ehsan Kamalloo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14970

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Self-Evolving Curriculum for LLM Reasoning Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, Ehsan Kamalloo Abstract Reinforcement learning (RL) has proven effective for fine-tuning large language models (LLMs), significantly enhancing their reasoning abilities in domains such as mathematics and code generation. A crucial factor influencing RL fine-tuning success is the training curriculum: the order in which training problems are presented. While random curricula serve as common baselines, they remain suboptimal; manually designed curricula often rely heavily on heuristics, and online filtering methods can be computationally prohibitive. To address these limitations, we propose Self-Evolving Curriculum (SEC) , an automatic curriculum learning method that learns a curriculum policy concurrently with the RL fine-tuning process. Our approach formulates curriculum selection as a non-stationary Multi-Armed Bandit problem, treating each problem category (e.g., difficulty level or problem type) as an individual arm. We leverage the absolute advantage from policy gradient methods as a proxy measure for immediate learning gain. At each training step, the curriculum policy selects categories to maximize this reward signal and is updated using the TD(0) method. Across three distinct reasoning domains: planning, inductive reasoning, and mathematics, our experiments demonstrate that SEC significantly improves models’ reasoning capabilities, enabling better generalization to harder, out-of-distribution test problems. Additionally, our approach achieves better skill balance when fine-tuning simultaneously on multiple reasoning domains. These findings highlight SEC as a promising strategy for RL fine-tuning of LLMs. 1 1 1 Our code will be publicly released soon. 1 Introduction Figure 1 : Curriculum matters. A deliberately poor (reverse) curriculum severely limits RL fine-tuning performance. Our proposed Self-Evolving Curriculum (SEC) significantly outperforms the standard random curriculum. See Sec. 3.1 for details. Reinforcement learning (RL) has emerged as a central technique for fine-tuning large language models (LLMs) [ 28 , 35 , 12 ] , significantly improving their reasoning capabilities. Recent advances demonstrate notable success, particularly in domains where verifying generation correctness is straightforward [ 27 ] , such as mathematics and code generation. By optimizing LLMs with rewards solely defined by verifiable outcomes, RL fine-tuning encourages the emergence of complex reasoning behaviors, including self-correction and back-tracking strategies [ 25 , 63 , 16 ] , that substantially enhance reasoning performance. A critical factor influencing the effectiveness of RL fine-tuning is the training curriculum [ 7 ] , i.e., the order in which training data is presented. Since online RL inherently depends on the policy model itself to produce high-quality training trajectories, aligning the curriculum with the model’s current learning progress is critical. Ideally, such an alignment enables the model to continually encounter problems that yield maximal learning outcomes [ 42 , 29 , 20 ] . To illustrate this point concretely, we conduct a controlled experiment using the Countdown game, 2 2 2 A puzzle game where players combine a given set of numbers using basic arithmetic operations to reach a target number. deliberately employing a suboptimal (reverse) curriculum, in which problems are arranged from hard to easy. As shown in Figure 1 , the resulting model performs poorly on the test set and exhibits minimal generalization to more challenging out-of-distribution (OOD) problems. In contrast, when trained with a baseline random curriculum, where problems of varying difficulty are drawn uniformly at random, the model demonstrates significantly improved generalization and overall task performance. Figure 2 : Overview of Self-Evolving Curriculum (SEC) . SEC dynamically adjusts the training curriculum according to the model’s current capabilities. During preprocessing, training data is partitioned into distinct categories (indicated by colors), e.g., by difficulty level or problem type. At each RL fine-tuning step: (1) The curriculum policy samples a training batch based on categories’ expected learning gains; (2) The LLM policy is updated using the sampled batch and the chosen RL algorithm; (3) Rewards for curriculum categories are computed using advantage values estimated by the RL algorithm; (4) The curriculum policy is updated accordingly, refining future data selection. Although random curriculum serves as a reasonable baseline, it naturally raises the question: Can we design more effective curriculum strategies ? Curriculum learning [ 7 ] addresses precisely this challenge by seeking to optimize the sequencing of training tasks, thereby enhancing learning efficiency and efficacy. Recent approaches to curriculum learning for RL fine-tuning typically involve either manually crafted curricula designed upfront [ 50 , 47 , 58 ] or dynamic online filtering based on on-policy samples [ 64 , 3 ] . However, manually designed curricula rely heavily on heuristics, demanding human intervention for new models or tasks; conversely, online filtering methods incur substantial computational overhead due to the continuous generation of additional on-policy samples. In this paper, we propose Self-Evolving Curriculum (SEC) (Figure 2 ), an automatic curriculum learning [ 39 ] approach for RL fine-tuning of LLMs. Our method adaptively learns a curriculum policy concurrently with the RL fine-tuning process, formulating curriculum selection as a non-stationary Multi-Armed Bandit (MAB) problem [ 51 , 49 , 31 ] . Each curriculum category (e.g., difficulty level or problem type) is treated as an individual arm, and the curriculum policy aims to select the arm that maximizes the learning outcomes. Specifically, we operationalize the concept of learning outcomes using the gradient norm, noting that, in policy gradient methods, the gradient norm is weighted by the absolute value of the advantage function. Leveraging this observation, we define the absolute advantage as the reward for each arm. At each RL training step, the curriculum is sampled according to the current MAB policy, which is subsequently updated on-the-fly using the reward obtained from the current training step via the TD(0) method [ 48 ] . Our experiments demonstrate that SEC significantly improves model reasoning capabilities across three distinct domains: planning, inductive reasoning, and mathematics, particularly improving generalization to challenging out-of-distribution problems. Compared to the standard random curriculum, SEC achieves substantial relative improvements, such as 13% on Countdown, 21% on Zebra puzzles, 22% on ARC-1D, and up to 33% on the AIME24 dataset. When fine-tuned simultaneously across multiple reasoning domains, SEC effectively balances performance across tasks, underscoring its strength as an automatic curriculum learning strategy for RL fine-tuning of LLMs. 2 Method In the context of RL fine-tuning, at each training step t , the curriculum policy selects a subset D_{t}\subseteq D from the training problem set D to be provided to the LLM. In our work, we consider scenarios where the training problems can be categorized into N distinct categories. This assumption simplifies the curriculum optimization problem into learning an expected return Q_{t}(c) that maps category c to a real-valued score (Sec. 2.1 ). The training batch is then constructed by first sampling categories according to the curriculum policy, followed by sampling problems uniformly within the categories. The goal of the curriculum policy is to maximize the LLM’s final task performance. However, directly evaluating such performance would require completing the entire RL fine-tuning process, while the curriculum policy is better to be updated along with the training steps. To resolve this, we introduce a locally measurable reward as a proxy objective for guiding the curriculum policy (Sec. 2.2 ). 2.1 Curriculum Selection as Multi-Armed Bandit Training datasets used for reasoning tasks can often be naturally decomposed into distinct categories. For example, if

Read the complete text