Diffusion Guided Adversarial State Perturbations in Reinforcement Learning
Xiaolin Sun, Feidi Liu, Zhengming Ding, Zizhan Zheng · 2025
Read Diffusion Guided Adversarial State Perturbations in Reinforcement Learning by Xiaolin Sun, Feidi Liu, Zhengming Ding, Zizhan Zheng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07701.txt.
Introduction
Xiaolin Sun, Feidi Liu, and Zhengming Ding expose a blind spot in defenses for vision-based learning agents. Small image tweaks can mislead reinforcement learning policies, so defenses hardened against pixel-norm attacks report strong results. Closer inspection shows those attacks barely change what the image means, even with generous budgets.
Diffusion-guided perturbations instead alter the state semantics while staying plausible. Such attacks stress defenses where it hurts: meaning, not pixels. Robustness claims built on weak attacks give false comfort for agents heading into open worlds. The study urges evaluation with attacks that match how perception actually fails.
Catalog record
| CREATOR | Xiaolin Sun, Feidi Liu, Zhengming Ding, Zizhan Zheng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 114 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07701
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Diffusion Guided Adversarial State Perturbations in Reinforcement Learning Xiaolin Sun, Feidi Liu, Zhengming Ding, Zizhan Zheng Abstract Reinforcement learning (RL) systems, while achieving remarkable success across various domains, are vulnerable to adversarial attacks. This is especially a concern in vision-based environments where minor manipulations of high-dimensional image inputs can easily mislead the agent’s behavior. To this end, various defenses have been proposed recently, with state-of-the-art approaches achieving robust performance even under large state perturbations. However, after closer investigation, we found that the effectiveness of the current defenses is due to a fundamental weakness of the existing l p l_{p} norm-constrained attacks, which can barely alter the semantics of image input even under a relatively large perturbation budget. In this work, we propose SHIFT, a novel policy-agnostic diffusion-based state perturbation attack to go beyond this limitation. Our attack is able to generate perturbed states that are semantically different from the true states while remaining realistic and history-aligned to avoid detection. Evaluations show that our attack effectively breaks existing defenses, including the most sophisticated ones, significantly outperforming existing attacks while being more perceptually stealthy. The results highlight the vulnerability of RL agents to semantics-aware adversarial perturbations, indicating the importance of developing more robust policies. Our code can be found at this GitHub Repo . 1 Introduction Reinforcement learning (RL) has seen significant advancements in recent years, becoming a key area of machine learning. RL’s ability to enable agents to learn optimal decision-making policies through interaction with dynamic environments have led to breakthroughs in various fields. Beginning from AlphaGo [ 44 ] , RL-based systems show the ability to surpass human performance in complex games. Beyond gaming, RL is driving innovations in robotics, self-driving cars [ 30 ] , and industrial automation, where agents learn to navigate, manipulate, and interact autonomously. However, RL is vulnerable to various types of attacks, such as reward and state perturbations, action space manipulations, and model inference and poisoning [ 26 ] . Recent studies have shown that an RL agent can be manipulated by perturbing its observation [ 24 , 61 ] and reward signals [ 25 ] , and a well-trained RL agent can be confounded by a malicious opponent behaving unexpectedly [ 17 ] . In particular, a malicious agent can subtly manipulate the observations of a trained RL agent, resulting in a significant drop in performance and cumulative reward [ 61 , 47 ] . Such attacks exploit vulnerabilities in the agent’s perception systems, including sensors and communication channels, without needing to cause obvious disruptions. This susceptibility to minor perturbations raises major concerns, particularly for RL applications in security-sensitive and safety-critical environments. Multiple defenses have been proposed to mitigate state perturbation attacks. Regularization-based methods like SA-MDP [ 61 ] and WocaR-MDP [ 36 ] improve robustness by smoothing policies and estimating worst-case rewards. CAR-DQN [ 35 ] further enhances robustness using the Bellman Infinity-error. Adversarial training methods including alternating training [ 60 ] and game-theoretical methods like PROTECTED [ 39 ] and GRAD [ 37 ] can potentially lead to more robust policies but are costly and do not scale to image-based inputs. More recently, diffusion-based methods, such as DMBP [ 57 ] and DP-DQN [ 46 ] , improve robustness by reconstructing the true states or modeling beliefs about them through denoising. These advanced defenses can withstand state-of-the-art attacks like PGD [ 61 ] , MinBest [ 24 ] , PA-AD [ 47 ] , and high-sensitivity direction attacks [ 32 ] . However, we found that current attacks share two major shortcomings when applied to agents with raw pixel images as input such as autonomous driving [ 30 ] and embodied AI [ 58 ] . First, with the exception of 32 , current attacks usually restrict a perturbed state to be within an \epsilon -ball of the true state, measured using an l_{p} norm, to avoid detection. However, this approach struggles with producing realistic perturbed input that is within the natural data distribution and further constrains the attacker’s search space for generating semantics-changing perturbations. While this problem has been considered in supervised settings [ 54 , 15 , 51 , 13 ] , to the best of our knowledge, 32 was the only prior study that identified this limitation for state perturbations in RL. However, the high-sensitivity direction attacks proposed in 32 adopt simple image transformations that mainly target changes in visually significant but not domain-specific semantics. Consequently, the perturbed states either can be denoised by diffusion-based defenses or are non-stealthy from human perspectives (see our evaluation results in Appendix D ). Second, previous attacks mainly focus on improving attack performance while ignoring the temporal dependencies across states, which is unique to RL. As a result, they often generate states that are inconsistent with the agent’s previous observations (either perturbed or not). Recently, 37 looked into this problem by considering two consecutive time steps, but it still utilized an l_{p} -norm constraint. On the other hand, Illusory attacks [ 16 ] require a perturbed trajectory to follow the same distribution as the normal trajectory, making it difficult to detect. However, this approach does not scale to high-dimensional image input. None of these attacks can easily modify the domain-specific semantics of the image input while keeping it realistic and plausible. The perturbed states generated by these attacks can be easily denoised with the help of a history-conditioned diffusion model. Figure 1 : A car approaches a crosswalk with a pedestrian ahead. The safe, optimal action is to brake. SHIFT-O removes the pedestrian from the agent’s observation, while SHIFT-I creates an imaginary trajectory suggesting the car has already crossed. Both mislead the agent into moving forward, resulting in a collision. With these two shortcomings in mind, we propose SHIFT ( S tealthy H istory-al I gned di F fusion a T tack), a novel semantics-aware and policy-agnostic attack method that goes beyond the traditional l_{p} norm constraint. Our approach is grounded in precise definitions of realistic states and three attack properties: semantic-altering, historically-aligned, and trajectory-faithful, which provide novel characterizations of semantics-aware and stealthy attacks in sequential decision-making. As these metrics are computationally expensive to evaluate, we provide practical methods to approximate them. Our main contribution is the development of a diffusion-based attack framework that utilizes classifier-free guidance to approximate history-aligned state generation, which is further improved using policy guidance to generate effective, realistic, and stealthy state perturbations. Using this novel guided diffusion approach, we propose two versions of SHIFT : SHIFT-O perturbs the image input conditioned on the actual history to immediately induce suboptimal actions, while SHIFT-I guides the agent toward an imagined trajectory that is self-consistent, but ultimately leads to poor performance when actions are executed in the real environment. We compare these two methods in Figure 1 , with a working example in the Freeway environment given in Figure 6 in Appendix C . SHIFT is policy-agnostic, capable of adapting to multiple victim policies without retraining, and applicable to both value-based and policy-based reinforcement learning methods. Comprehensive evaluations show that it can break all known defenses, lower agents’ cumulative reward in various environments by more than 50%, while being stealthier than prior attacks, as shown by lower reconstruction error, Wasserstein-1 distance,