Denoising-based Contractive Imitation Learning

Macheng Shen, Jishen Peng, Zefang Huang · 2025

Read Denoising-based Contractive Imitation Learning by Macheng Shen, Jishen Peng, Zefang Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15918.txt.

Introduction

Macheng Shen and co-authors steady imitation learning with a denoising trick in a 2025 paper. Robots that learn by copying experts drift off course as small errors pile up, a snag called covariate shift. Past fixes ask for more expert time, world models, or hard training duels.

Their method, DeCIL, trains a dynamics model to guess the next state and a joint denoising net to clean that guess and pick the act. Proofs show the denoiser acts as a local shrinking map, which trims error growth step by step. It drops into current imitation frames with no extra expert data, and tests show higher success under noise.

Catalog record

CREATORMacheng Shen, Jishen Peng, Zefang Huang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-15918

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Denoising-based Contractive Imitation Learning Macheng Shen, Jishen Peng, Zefang Huang Abstract A fundamental challenge in imitation learning is the covariate shift problem. Existing methods to mitigate covariate shift often require additional expert interactions, access to environment dynamics, or complex adversarial training, which may not be practical in real-world applications. In this paper, we propose a simple yet effective method DeCIL , De noising-based C ontractive I mitation L earning, to mitigate covariate shift by incorporating a denoising mechanism that enhances the contraction properties of the state transition mapping. Our approach involves training two neural networks: a dynamics model f f that predicts the next state from the current state, and a joint state-action denoising policy network d d that refines this state prediction via denoising and outputs the corresponding action. We provide theoretical analysis showing that the denoising network acts as a local contraction mapping, reducing the error propagation of the state transition and improving stability. Our method is simple to implement and integrates seamlessly with existing imitation learning frameworks without requiring additional expert data or complex modifications to the training procedure. Empirical results demonstrate that our approach effectively improves success rate of various imitation learning tasks under noise perturbation. Code can be viewed in https://github.com/MachengShen/Stable-BC . 1 Introduction Imitation learning enables agents to acquire complex behaviors by learning from expert demonstrations ( Pomerleau 1989 ; Argall et al. 2009 ) . It has been successfully applied in robotics ( Billard et al. 2008 ) , autonomous driving ( Bojarski et al. 2016 ) , and game playing ( Silver et al. 2016 ) . However, a fundamental challenge in imitation learning is the covariate shift problem ( Ross & Bagnell 2010 ; Ross et al. 2011 ) , where discrepancies between the training and execution state distributions lead to compounding errors. The learned policy may encounter states during execution that were not represented in the training data, resulting in poor generalization and degraded performance. Existing methods to mitigate covariate shift often require additional expert interaction, access to environment dynamics, or complex training procedures, which may not be practical in real-world applications. In this paper, we propose a simple yet effective approach to mitigate covariate shift by enhancing the contraction properties of state transitions through a denoising mechanism. Our method involves training two neural networks: 1. Dynamics Model f : Predicts the next state \hat{x}_{t+1} given the current state x_{t} . 2. Denoising Policy Network d : Takes the current state x_{t} and the predicted next state \hat{x}_{t+1} to output a refined next state \tilde{x}_{t+1} and the corresponding action \hat{a}_{t} . Our key insight is that by incorporating a denoising step, we can reduce the Lipschitz constant of the state transition mapping, effectively making it a local contraction mapping. This reduces the impact of prediction errors, preventing them from compounding over time. The main advantages of our approach are its simplicity and compatibility with existing imitation learning frameworks. It requires only access to expert demonstrations and does not necessitate additional expert interaction or complex training procedures. Moreover, our method can be easily integrated into standard training pipelines and can complement other techniques to further improve performance. 1.1 Paper Organization The rest of the paper is organized as follows: In Section 2 , we discuss related works and position our approach in the context of existing methods. Section 3 presents our method in detail. In Section 3.7.1 , we provide the theoretical analysis demonstrating the contraction properties of our approach. Section 4 presents empirical results validating our method on benchmark tasks. Finally, we conclude in Section 6 . 2 Related Works Imitation learning aims to learn policies that mimic expert behavior using demonstration data ( Argall et al. 2009 ) . Behavioral cloning (BC) ( Pomerleau 1989 ) treats imitation learning as supervised learning, training a policy to map states to actions directly from expert demonstrations. However, BC suffers from the covariate shift problem because the learned policy may encounter states during execution that are not represented in the training data, leading to compounding errors. Several methods have been proposed to mitigate covariate shift: Interactive Expert Queries. Methods like DAgger ( Ross et al. 2011 ) and its variants ( Laskey et al. 2017 ; Sun et al. 2017 ) involve querying the expert for corrective actions during the agent’s own state distribution. While effective, these approaches require ongoing access to the expert, which may not be feasible in many practical scenarios. Adversarial Imitation Learning. Approaches such as GAIL ( Ho & Ermon 2016 ) and AIRL ( Fu et al. 2018 ) formulate imitation learning within a generative adversarial framework, where a discriminator distinguishes between expert and agent behaviors. These methods aim to align the agent’s trajectory distribution with that of the expert ( Blondé et al. 2022 ; Ghasemipour et al. 2020 ) . However, they often rely on querying the environment during training, which can be infeasible in many imitation learning scenarios where access to the environment is limited or costly. Additionally, adversarial training introduces challenges such as instability and sensitivity to hyperparameters ( Wiatrak et al. 2019 ) . Data Augmentation. Techniques that augment the training data have been explored to enhance robustness. For instance, Ke et al. 2021a ; Jiang et al. 2024 propose generating synthetic data using learned dynamics models, while Florence et al. 2019 ; Zhou et al. 2023 ; Spencer et al. 2021 ; Hoque et al. 2024 leverage domain-specific invariances to create augmented samples. However, these methods often rely on additional assumptions, such as knowledge of system invariances or access to accurate dynamics models, which may not be available. Stability and Contractive Policies. Incorporating stability properties into policy learning has gained attention as a way to enhance robustness. Blocher et al. 2017 and Ravichandar et al. 2017 focus on learning stable dynamical systems that guarantee convergence to desired states. Beik-Mohammadi et al. 2024 ; Abyaneh et al. 2024 introduce contractive dynamical systems to ensure exponential convergence and improve out-of-sample recovery. While these methods provide theoretical guarantees, they often involve complex constrained optimization, specific architectures or assume accessibility to additional low-level controller, which can limit their scalability and practicality. Simple and Integrable Approaches. Recent works have highlighted the need for methods that are simple to implement and can be easily integrated with existing frameworks. ( Ke et al. 2021b ) propose a simple noise-perturbing mechanism to alleviate covariate-shift. ( Mehta et al. 2024 ) propose Stable-BC, which reguluarizes the eigenvalues of the Jacobian of the closed-loop dynamics to achieve stability. However, their approach requires an accurate dynamics model to avoid trading-off performance for stability. Besides, computing Jacobian is intractable for high-dimensional state space, which restricts the applicability of their approach. Our Contribution. In contrast to these approaches, our method provides a simple yet effective solution to covariate shift by incorporating a denoising mechanism. By training a denoising policy network with a denoising objective, we encourage contraction in the state transition mapping without requiring complex constraints, additional expert interaction, or access to environment dynamics. Our approach is easy to implement, requires minimal additional assumptions beyond access to expert data, and can be seamlessly integrated into existing imitation learning pipelines. 3 Method 3.1 Problem Formulation

Read the complete text