PNAct: Crafting Backdoor Attacks in Safe Reinforcement Learning
Weiran Guo, Guanjun Liu, Ziyuan Zhou, Ling Wang · 2025
Read PNAct: Crafting Backdoor Attacks in Safe Reinforcement Learning by Weiran Guo, Guanjun Liu, Ziyuan Zhou, Ling Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00485.txt.
Introduction
Weiran Guo and co-authors wrote this security paper in 2025 on safe reinforcement learning. Such agents learn by reward but must also keep a cost score within bounds, so they act safely. The team shows this safety can be hacked by a backdoor: hidden training tricks that make the agent act unsafely when a trigger shows. They name their attack frame PNAct.
The frame plants the backdoor with two kinds of samples. Good samples show acts to copy, and bad samples mark acts to shun, and the mix steers the agent's hidden switch. The team proves traits of the attack and builds a method to run it. Tests with their new scoring metrics show the attack works. They publish code and urge the field to treat safe learning as a target, not a given.
Catalog record
| CREATOR | Weiran Guo, Guanjun Liu, Ziyuan Zhou, Ling Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-00485
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PNAct: Crafting Backdoor Attacks in Safe Reinforcement Learning Weiran Guo, Guanjun Liu, Ziyuan Zhou, Ling Wang Abstract Reinforcement Learning (RL) is widely used in tasks where agents interact with an environment to maximize rewards. Building on this foundation, Safe Reinforcement Learning (Safe RL) incorporates a cost metric alongside the reward metric, ensuring that agents adhere to safety constraints during decision-making. In this paper, we identify that Safe RL is vulnerable to backdoor attacks, which can manipulate agents into performing unsafe actions. First, we introduce the relevant concepts and evaluation metrics for backdoor attacks in Safe RL. It is the first attack framework in the Safe RL field that involves both Positive and Negative Action sample (PNAct) is to implant backdoors, where positive action samples provide reference actions and negative action samples indicate actions to be avoided. We theoretically point out the properties of PNAct and design an attack algorithm. Finally, we conduct experiments to evaluate the effectiveness of our proposed backdoor attack framework, evaluating it with the established metrics. This paper highlights the potential risks associated with Safe RL and underscores the feasibility of such attacks. Our code and supplementary material are available at https://github.com/azure-123/PNAct. 1 Introduction Reinforcement learning (RL), as an important branch of artificial intelligence, is primarily focused on training an agent to develop an optimal policy that maximizes task gains through interactions with the environment. The metric used to evaluate task gains at every time step is the reward signal provided by the environment during the interaction process Shi et al. 2023 ; Chen et al. 2024 ; Yang et al. 2023 . The greater the reward, the better the agent’s action at that time step. However, the optimal policy in RL is often aggressive and may select actions with potential safety risks when making decisions, failing to ensure that the agent adheres to safety constraints. Therefore, existing research has proposed Safe Reinforcement Learning (Safe RL) based on traditional RL Heger 1994 . Safe RL introduces an auxiliary cost signal during the training process to measure the policy’s compliance with safety constraints. In this context, the agent needs to simultaneously maximize cumulative rewards and control cumulative costs, ensuring task completion while complying with safety constraints. Existing solution methods for Safe RL include primal-dual-based methods Bhatnagar and Lakshmanan 2012 ; Fujimoto et al. 2019 , uncertainty-aware methods Fujimoto et al. 2019 ; Chen et al. 2021 , and various other approaches Achiam et al. 2017 ; Liu et al. 2022 ; Zhang et al. 2020 ; Yang et al. 2020 . However, beyond environmental safety constraints, RL faces other security challenges, such as adversarial attacks and backdoor attacks. While these issues are serious and deserve attention, research in this area remains limited. Existing studies Liu et al. 2023a ; Liu et al. 2023b have shown that Safe RL can be vulnerable to adversarial attacks, which can lead to excessively high violation costs for safety constraints. These studies have also proposed corresponding defense methods. In contrast, backdoor attacks present a more challenging defense problem than adversarial attacks. Backdoor attacks are inherently more covert, as they involve injecting specific trigger conditions during training. Once triggered, these conditions cause the agent to take malicious or unsafe actions while the agent appears to behave normally under regular circumstances. Due to their stealthy, backdoor attacks are more difficult to defend against. If effectively implemented on Safe RL, backdoor attacks could significantly compromise the agent’s decision-making safety and greatly increase the difficulty of implementing effective defenses. A recent study Jiang et al. 2024 proposed a safety reinforcement learning backdoor based on Signal Temporal Logic (STL), demonstrating the vulnerability of Safe RL to such attacks. To the best of our knowledge, this is the first exploration of backdoor attacks in the context of Safe RL. However, this approach has certain limitations. The proposed backdoors require integrating logic parsers with RL frameworks and using different specifications tailored to specific scenarios, which limits their general applicability. To address this research gap, this paper makes the following contributions: • We define the concepts related to backdoor attacks in Safe RL and propose relevant metrics to evaluate the effectiveness and stealthiness of these attacks. • We propose a backdoor attack framework for Safe RL, named PNAct, which leverages positive and negative action samples along with a loss function modification technique. This framework enables the agent to take unsafe and risky actions when the environment reaches specific states. • We apply the proposed attack framework to safety-constrained environments and evaluate its performance using the backdoor metrics for Safe RL, demonstrating the effectiveness of our approach. Unlike previous approaches, our attack framework does not affect the rewards obtained by the agent, making it less likely to be detected. However, it increases the cost of violating safety constraints, leading to riskier decision-making by the agent. Moreover, it can be applied to a variety of environments without requiring additional scenario-specific designs. 2 Related Work 2.1 Safe RL Safe RL focuses on maximizing rewards while controlling costs. A popular method is the primal-dual approach, which introduces a Lagrange multiplier to penalize constraint violations, unifying reward and cost optimization. This approach serves as the foundation for our backdoor attack framework. Early studies manually selected the Lagrange multiplier Borkar 2005 ; Bhatnagar and Lakshmanan 2012 ; As et al. 2022 , but this often led to suboptimal policies. To improve, training can dynamically update policy parameters and the multiplier to balance rewards and costs Chow et al. 2018 ; Liang et al. 2018 . Additionally, PID control Stooke et al. 2020 helps stabilize training and reduce oscillations caused by phase shifts between the constraint function and multiplier. 2.2 Backdoor Attacks in RL Backdoor attacks in RL resemble those in neural network classification, where an agent is manipulated to perform attacker-specified actions in certain states while behaving normally in other states. Most attacks decrease rewards. TrojDRL Kiourti et al. 2020 , uses poisoning methods and provides various strategies based on attack strength and target specificity. Building on TrojDRL, specialized methods have emerged: BACKDOORL Wang et al. 2021 targets adversarial RL by triggering backdoors via adversarial action sequences; BadRL Cui et al. 2024 employs sparse attacks to maximize success with minimal steps; MARLNet Chen et al. 2023b focuses on multi-agent scenarios using imperceptible triggers and reward manipulation to degrade performance. These attacks prioritize reward reduction without addressing safety constraint violations. Recent research introduces the first backdoor attack in Safe RL using STL Jiang et al. 2024 , but its effectiveness is limited in complex, high-dimensional, or stochastic environments due to challenges in defining and verifying STL specifications and high computational costs. In contrast, our proposed method generalizes better across diverse environments. 3 Preliminary 3.1 Constrained Markov Decision Process (CMDP) The traditional Markov Decision Process (MDP) in reinforcement learning is a tuple that only considers the reward signal, whereas the CMDP Altman 1998 defines a cost signal. It is represented as \langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\mathcal{C},\rho,\gamma,\rangle , where \mathcal{S} is the set of all possible states in the environment, and \mathcal{A} is the set of actions the agent can take. \mathcal{P}(s_{t+1}|s_{t},a_{t}):\mathcal{S}\times\mathcal{A}\times\mathcal{S}\to[0,1] is the probability that, at time step t , the environment transitions from state s_{t}\in\mathcal{S} to state s_{t+1}\in\mathcal{S} after the agent takes