ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning

Zeyuan Liu, Zhihe Yang, Jiawei Xu, Rui Yang, Jiafei Lyu, Baoxiang Wang, Yunjian Xu, Xiu Li · 2025

Read ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning by Zeyuan Liu, Zhihe Yang, Jiawei Xu, Rui Yang, Jiafei Lyu, Baoxiang Wang, Yunjian Xu, Xiu Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23871.txt.

Introduction

Liu and co-authors clean corrupted training logs so offline reinforcement learning agents can still learn from them. Real logs from sensors and humans carry noise in states, actions, and rewards at once, and earlier fixes handle only some of that. Diffusion models denoise well but tend to memorize the noise itself.

Their 2025 method first trains a diffusion variant on partly corrupted data with theoretical backing. Its noise predictions then sort clean rows from dirty ones. A standard diffusion model trained on the clean subset repairs the dirty rows. The cleaned data plugs into any offline learning algorithm. Across robot and manipulation benchmarks, the pipeline reached top results under many noise settings.

Catalog record

CREATORZeyuan Liu, Zhihe Yang, Jiawei Xu, Rui Yang, Jiafei Lyu, Baoxiang Wang, Yunjian Xu, Xiu Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE84 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23871

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning Zeyuan Liu, Zhihe Yang, Jiawei Xu, Rui Yang, Jiafei Lyu, Baoxiang Wang, Yunjian Xu, Xiu Li Abstract Real-world datasets collected from sensors or human inputs are prone to noise and errors, posing significant challenges for applying offline reinforcement learning (RL). While existing methods have made progress in addressing corrupted actions and rewards, they remain insufficient for handling corruption in high-dimensional state spaces and for cases where multiple elements in the dataset are corrupted simultaneously. Diffusion models, known for their strong denoising capabilities, offer a promising direction for this problem—but their tendency to overfit noisy samples limits their direct applicability. To overcome this, we propose A mbient D iffusion- G uided Dataset Recovery ( ADG ), a novel approach that pioneers the use of diffusion models to tackle data corruption in offline RL. First, we introduce Ambient Denoising Diffusion Probabilistic Models (DDPM) from approximated distributions, which enable learning on partially corrupted datasets with theoretical guarantees. Second, we use the noise-prediction property of Ambient DDPM to distinguish between clean and corrupted data, and then use the clean subset to train a standard DDPM. Third, we employ the trained standard DDPM to refine the previously identified corrupted data, enhancing data quality for subsequent offline RL training. A notable strength of ADG is its versatility—it can be seamlessly integrated with any offline RL algorithm. Experiments on a range of benchmarks, including MuJoCo, Kitchen, and Adroit, demonstrate that ADG effectively mitigates the impact of corrupted data and improves the robustness of offline RL under various noise settings, achieving state-of-the-art results. 1 Introduction Offline reinforcement learning (RL) has emerged as a prominent paradigm for learning decision-making policies from offline datasets ( 20 , 11 ) . Existing approaches can be broadly categorized into MDP-based methods ( 11 , 10 , 18 , 19 , 4 , 5 , 12 ) and non-MDP methods ( 6 , 16 , 29 ) . However, due to the data-dependent nature of offline RL, it encounters significant challenges when dealing with offline data subjected to random noise or adversarial corruption ( 42 , 41 , 22 , 36 , 37 ) . Such disturbances can cause substantial performance degradation or result in a pronounced deviation from the intended policy objectives. Therefore, ensuring robust policy learning is crucial for offline RL to function effectively in real-world scenarios. Several previous studies have focused on the theoretical properties and certification of offline RL under corrupted data ( 43 , 40 , 33 , 7 , 39 ) . Empirical efforts have resulted in the development of uncertainty-weighted algorithms utilizing Q-ensembles ( 40 ) and robust value learning through Huber loss and quantile Q estimators [ 37 ] . Additionally, sequence modeling techniques have been applied to mitigate the effects of data corruption while iteratively refining noisy actions and rewards in the dataset ( 34 ) . However, as noted in ( 34 ) , recovering observations remains challenging due to the high dimensionality. Figure 1: Overview of the training processes for the detector and denoiser (left) and the dataset recovery process (right) in the proposed ADG method. In recent years, diffusion-based generative models ( 30 , 14 , 31 ) have gained considerable attention for their ability to effectively model complex data distributions, making them increasingly important in offline RL ( 15 , 3 , 23 ) . One promising application of these models is their usage in reducing or mitigating noise in data, thanks to their inherent denoising capabilities. For instance, DMBP ( 38 ) proposed a diffusion-based framework aimed at minimizing noise in observations during the testing phase. Despite its success, this approach is specifically designed to work with clean training data and only addresses data perturbations during testing. It cannot manage perturbations in the training dataset, as current diffusion models typically assume the dataset is entirely clean or has a consistent noise distribution across all data points ( 1 , 8 ) . Therefore, naive diffusion methods can encounter challenges, such as overfitting to noise data, when applied to partially corrupted data during training. To gain deeper insight into which aspects of corrupted datasets degrade the performance of offline RL algorithms, we evaluate several offline RL methods on three types of datasets: a clean dataset, a partially corrupted dataset, and a filtered dataset, which is created by removing the noisy portions from the corrupted data, as illustrated in Figure 2 (a–c). Surprisingly, We find that current offline RL algorithms fail to fully restore their performance on the filtered dataset, particularly when the dataset size is limited. These results imply that the loss of critical sequential information plays a key role in performance degrading of decision-making. Additionally, while sequence modeling methods ( 6 , 34 ) exhibit robustness against data corruption, they fail to function when faced with incomplete trajectories. This finding indicates that simply filtering out noisy samples is insufficient for handling corrupted datasets, underscoring the importance of recovering corrupted data. Figure 2: The performance of baseline algorithms (CQL, IQL, RIQL, DT, and RDT) is evaluated under four dataset conditions: Clean (original dataset without corruption), Noised (dataset corrupted using Random State Attack as described in Appendix C.1 ), Filtered (Noised dataset with corrupted samples removed), and Recovered (dataset recovered using ADG). The comparison results show the average normalized scores across three MuJoCo tasks ( halfcheetah , hopper , and walker2d ) using the “medium-replay-v2” datasets. We include the results for both the 100% dataset and 10% dataset settings. ADG effectively restores performance close to the results on clean datasets. To address this issue, we introduce the first diffusion-based denoising framework for handling corruption robust offline RL. Our approach recovers clean data purely from the corrupted dataset, without requiring any external information or supervision. We name this novel three-stage diffusion-based method A mbient D iffusion- G uided Dataset Recovery ( ADG ). The detailed diagram is presented in Figure 1 . In the first stage, we introduce Ambient Denoising Diffusion Probabilistic Models (DDPM) from approximated distributions , which enable diffusion training on partially corrupted datasets with theoretical guarantees. In the second stage, leveraging the noise-prediction property of the well-trained ambient DDPM, we identify the corrupted data within the dataset. The remaining clean data is then used to train a denoiser within the framework of standard DDPM. Finally, in the third stage, we apply the standard DDPM to refine the previously identified corrupted data. The corrected data is combined with the clean data to form a high-quality dataset, which is subsequently used for offline RL training. We find that ADG is overall competitive with the ideal case of perfectly filtering out all noisy samples for MDP-based algorithms and significantly outperforms the filtering method for sequence modeling methods, as shown in Figure 2 (d). Additionally, we provide a comprehensive analysis of ADG’s performance under both random and adversarial data corruption scenarios, examining various levels of data availability, whether full or limited dataset size, as described in Section 5 . Our empirical studies demonstrate that ADG consistently enhances the performance of all baseline algorithms, achieving a remarkable overall improvement. These findings indicate that ADG is versatile and compatible with any offline RL approach, including

Read the complete text