Causally Aligned Curriculum Learning

Mingxuan Li and Junzhe Zhang and Elias Bareinboim · 2025

Read Causally Aligned Curriculum Learning by Mingxuan Li and Junzhe Zhang and Elias Bareinboim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16799.txt.

Introduction

Mingxuan Li, Junzhe Zhang, and Elias Bareinboim present causally aligned curriculum learning in two thousand twenty-five. Curriculum learning trains agents on easy tasks before a hard target. It leans on shared optimal rules across the task chain. Hidden confounders can break that sharing in real settings.

The team views the snag through causal lenses. They derive a graph rule that marks source tasks as causally aligned. Aligned tasks do keep the shared rule trait. An algorithm then builds such a curriculum from causal knowledge of the target. Tests on confounded tasks, some with pixel inputs, back the method.

Catalog record

CREATORMingxuan Li and Junzhe Zhang and Elias Bareinboim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16799

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Causally Aligned Curriculum Learning Mingxuan Li and Junzhe Zhang and Elias Bareinboim Abstract A pervasive challenge in Reinforcement Learning (RL) is the “curse of dimensionality” which is the exponential growth in the state-action space when optimizing a high-dimensional target task. The framework of curriculum learning trains the agent in a curriculum composed of a sequence of related and more manageable source tasks. The expectation is that when some optimal decision rules are shared across source tasks and the target task, the agent could more quickly pick up the necessary skills to behave optimally in the environment, thus accelerating the learning process. However, this critical assumption of invariant optimal decision rules does not necessarily hold in many practical applications, specifically when the underlying environment contains unobserved confounders . This paper studies the problem of curriculum RL through causal lenses. We derive a sufficient graphical condition characterizing causally aligned source tasks, i.e., the invariance of optimal decision rules holds. We further develop an efficient algorithm to generate a causally aligned curriculum, provided with qualitative causal knowledge of the target task. Finally, we validate our proposed methodology through experiments in discrete and continuous confounded tasks with pixel observations. 1 Introduction As Roma was not built in one day, learning to achieve a complex task (e.g., cooking, driving) directly can be challenging. Instead, the human learning process is scaffolded with incremental difficulty to support acquiring progressively advanced knowledge and skills. The idea of training with increasingly complex tasks, known as curriculum learning, has been applied in reinforcement learning when Selfridge et al. 1985 used a carefully curated sequence of tasks to train agents to solve a modified Cart Pole system. In recent years, there has been a growing interest in automatically generating curricula tailored to the agent’s current capabilities, which opens up a new venue called “Automatic Curriculum Learning” ( Portelas et al. 2020 ) . An automatic curriculum generator requires two components: an encoded task space and a task characterization function ( Narvekar et al. 2020 ; Wang et al. 2020 ) . Task space encoding is often a bijective function that maps a task to a low dimensional vector ( Parker-Holder et al. 2022 ; Klink et al. 2022 ; Florensa et al. 2018 ; Jiang et al. 2021 ; Portelas et al. 2019 ; Wang et al. 2019 ; Wang et al. 2020 ; Cho et al. 2023 ; Huang et al. 2022a ) . A proper task space encoding lays the foundation of a reasonable task characterization function measuring the fitness of tasks ( Florensa et al. 2018 ; Dennis et al. 2020 ; Andreas et al. 2017 ; Sukhbaatar et al. 2018 ; Jiang et al. 2021 ) . New training tasks, called source tasks, are generated by changing the target task’s state space or parameters of transition functions in the encoded task space. A system designer then determines in which order the agent should be trained in these source tasks, following the task characterization function. The set of generated source tasks and the training order defined upon this set defines a curriculum for the learning agent. Please see Appendix G for more related work. (a) Misaligned Source Task (b) Aligned Source Task (c) Aligned Curriculum Figure 1: Examples of ( 1(a) ) full episode of a misaligned source task that intervenes in the box color, ( 1(b) ) full episode of an aligned source task that only changes the initial box location, and ( 1(c) ) an aligned curriculum where none of the source tasks intervenes in the box’s color. While impressive, most curriculum RL methods described so far rely on the assumption that generated source tasks are aligned with the target. Consequently, the agent could pick up some valuable skills by training in such source tasks, allowing it to behave optimally in certain situations in the target environment. However, this critical assumption does not necessarily hold in many real-world decision-making settings. For concreteness, consider a modified Sokoban game shown in Figure 1 inspired by Schrader 2018 where an unobserved confounder U_{t} randomly determines the box color C_{t} ( 0 for yellow, 1 for blue) at every time step t . The agent receives a positive reward Y_{t} only when it pushes the box to the goal state when the box color appears yellow ( U_{t}=0 ); otherwise, it gets penalized ( U_{t}=1 ). We apply several state-of-the-art curriculum generators that construct source tasks by fixing the box color to yellow or blue, including ALP-GMM ( Portelas et al. 2019 ) , PLR ( Jiang et al. 2021 ) , Goal-GAN ( Florensa et al. 2018 ) , and Currot ( Klink et al. 2022 ) . Figure 1(a) shows an example of the generated source tasks. We evaluate agents’ performance trained by those generated curricula and compare it with the one directly trained in the target task. Surprisingly, simulation results shown in Figure 2 reveal that agents trained by the curricula failed to learn to push the yellow box to the destination. This suggests source tasks generated by intervening in the box color are misaligned; that is, training in these source tasks harms the agents’ target task performance. Figure 2: The average performance of curriculum generators. Several observations follow from the Sokoban example. (1) A curriculum designer generates source tasks by modifying the data-generating mechanisms in the target tasks. (2) Such modifications could lead to a shift in system dynamics between the target task and source tasks. When this distribution shift is significant, training in source tasks may harm the agent’s learning. (3) The agent must avoid misaligned source tasks to achieve optimal learning performance. There exist methods attempting to address the challenges of misaligned source tasks leveraging a heuristic similarity measure between the target and source tasks ( Svetlik et al. 2017 ; Silva & Costa 2018 ) . Yet, a systematic and theoretically justified approach for exploiting other types of knowledge, e.g., qualitative, about the target task is missing. This paper aims to address the challenges of misaligned source tasks in curriculum generation by exploring causal relationships among variables present in the underlying environment. To realize this, we formalize curriculum learning in the theoretical framework of structural causal models (SCMs) ( Pearl 2009 ) . This formulation allows us to characterize misaligned source tasks by examining the structural invariance across the optimal policies obtained from the target and source tasks. More specifically, our contributions are summarized as follows. (1) We derive a sufficient graphical condition determining potentially misaligned source tasks. (2) We develop efficient algorithms for detecting misaligned source tasks and constructing source tasks that are guaranteed to align with the target task. (3) We introduce a novel augmentation procedure that enables state-of-the-art curriculum learning algorithms to generate aligned curricula to accelerate the agent’s learning. Finally, we validate the proposed framework through extensive experiments in various decision-making tasks. 1.1 Preliminaries This section introduces necessary notations and definitions that will be used throughout the discussion. We use capital letters ( X ) to denote a random variable, lowercase letters ( x ) to represent a specific value of the random variable, and \Omega(\cdot) to denote the domain of a random variable. We use bold capital letters

Read the complete text