Online Poisoning Attack Against Reinforcement Learning under Black-box Environments
Jianhui Li, Bokang Zhang, Junfeng Wu · 2024
Read Online Poisoning Attack Against Reinforcement Learning under Black-box Environments by Jianhui Li, Bokang Zhang, Junfeng Wu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00797.txt.
Introduction
Jianhui Li and co-authors build an on-line taint strike on learn agents when the world stays a black box. The work dates to 2024. A foe can warp teach data to bend an agent toward a bad plan. Past strikes knew the world's rules; here the foe does not.
The strike warps both prize scores and step shifts. The task is cast as a bound math hunt. Since shift odds are dark to the foe, slopes come from loose test draws. A fee trick and a two-tier recast dodge a clash in the math. A maze world showed the strike can bend the agent's plan.
Catalog record
| CREATOR | Jianhui Li, Bokang Zhang, Junfeng Wu |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00797
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Online Poisoning Attack Against Reinforcement Learning under Black-box Environments Jianhui Li, Bokang Zhang, Junfeng Wu Abstract This paper proposes an online environment poisoning algorithm tailored for reinforcement learning agents operating in a black-box setting, where an adversary deliberately manipulates training data to lead the agent toward a mischievous policy. In contrast to prior studies that primarily investigate white-box settings, we focus on a scenario characterized by unknown environment dynamics to the attacker and a flexible reinforcement learning algorithm employed by the targeted agent. We first propose an attack scheme that is capable of poisoning the reward functions and state transitions. The poisoning task is formalized as a constrained optimization problem, following the framework of Ma et al. (2019) . Given the transition probabilities are unknown to the attacker in a black-box environment, we apply a stochastic gradient descent algorithm, where the exact gradients are approximated using sample-based estimates. A penalty-based method along with a bilevel reformulation is then employed to transform the problem into an unconstrained counterpart and to circumvent the double-sampling issue. The algorithm’s effectiveness is validated through a maze environment. keywords Data poisoning attack; reinforcement learning; bilevel optimization 1 Introduction Reinforcement learning (RL) algorithms have gained substantial popularity in recent years, finding applications across a variety of fields such as robotic manipulation ( Gu et al. (2016) ), recommendation systems ( Chen et al. (2019) ), and autonomous vehicles ( O’Kelly et al. (2018) ). Despite their success, these algorithms typically depend on vast amounts of collected data for effective agent training. This reliance presents a significant vulnerability: adversaries can disrupt the training process by injecting poisoned data, thereby steering the agent towards adopting potentially malicious policies. A striking example is Tay, a conversational AI chatbot developed by Microsoft that incorporates adaptive learning capabilities. Through the coordinated efforts of certain malicious Twitter users, Tay was manipulated into producing racist and antisocial content ( Neff (2016) ). The paper Kiourti et al. (2020) was one of the pioneering works to investigate the design of data poisoning attacks against reinforcement learning (RL) agents. It introduced a man-in-the-middle attack (MIMA) framework that alters state data. In this framework, the adversary intercepts and manipulates the data from the environment to the agent, effectively directing the agent to adopt a predefined target policy. Subsequent studies have expanded this line of research to encompass scenarios with limited attack budgets Rakhsha et al. (2021) ; Zhang et al. (2020) , sparse attacks with stealthy awareness Cui et al. (2024) , and multi-agent environments Wu et al. (2023) ; Liu and Lai (2023) . However, a significant limitation of the majority of these approaches is their assumption of prior knowledge about reward and transition models. This assumption is less realistic in the context of RL, as RL typically operates in an online setting. In this paper, we focus on an online MIMA in a black-box context, wherein the adversary operates without prior knowledge of environment dynamics. The attacker is capable of modifying the reward and transitioned state data. To enhance the stealthiness of attack, we formalize the task as an optimization problem aimed at minimizing the deviation from the original MDP. Furthermore, the modified transitioned states are restricted to those that are reachable in the original environment. This attack is termed online because it takes place simultaneously with the agent-environment interaction over time. Our main contributions are summarized as follows: 1. In contrast to the previous transition probability poisoning algorithms that primarily focus on white-box settings (such as Rakhsha et al. (2020) ), we propose a more practical poisoning framework that operates effectively in scenarios where the environment dynamics are unknown to the attacker and the targeted agent employs a flexible reinforcement learning algorithm. 2. We develop an iterative optimization algorithm for seeking an optimal poisoning tactics. This method employs a penalty-based approach combined with a bilevel reformulation to avoid the so-called double-sampling issue. In addition, we leverage stochastic gradient descent technique in bilevel optimization, enabling an efficient online poisoning execution. Notation : The indicator function \mathbf{1}(\cdot) outputs 1 if the event of interest is true and 0 otherwise. The cardinality of a finite set \mathcal{A} is denoted as |\mathcal{A}| . To differentiate between variables related to the attacker and those related to the RL agent, we use overlined symbols to represent the attacker’s variables. 1.1 Related Works Existing research on data poisoning attacks against RL agents have been studied under different contexts. Black-box vs. White-box Environments . In white-box scenario, attackers are usually assumed to have prior knowledge of the environment dynamics, while attackers in black-box settings posit that such information is unavailable to them. Most existing studies ( Kiourti et al. (2020) ; Ma et al. (2019) ; Rakhsha et al. (2020) ; Foley et al. (2022) ) focus primarily on white-box scenarios, which may be practical for supervised learning victims but are less realistic in the context of reinforcement learning. The paper Rakhsha et al. (2021) investigates an online attack, in which the environment dynamics is unknown. The authors categorize the attack into two phases: the exploration phase and the attack phase. However, establishing the transition criteria between the two phases poses a significant challenge. The work Xu and Singh (2023) describes an approach where the attacker adjusts the rewards of actions based on their proximity to the target action. Both of these methods employ heuristic strategies. Poisoning Context . In the realm of adversarial attacks, attackers are assumed to be capable of poisoning various components of data. Research has explored the manipulation of state information Ashcraft and Karra (2021) and action poisoning Liu and Lai (2021) . However, a significant number of studies concentrate on altering reward data ( Ma et al. (2019) ; Zhang et al. (2020) ; Wu et al. (2023) ; Rangi et al. (2022) ), as rewards are typically manually designed and tend to be less sensitive to minor changes. Some studies also explore poisoning of both reward and transition probabilities ( Rakhsha et al. (2020) ; Xu et al. (2022) ). Notably, Rakhsha et al. (2020) examines a white-box attack in which transition probabilities are assumed to be known. Xu et al. (2022) presents a method for poisoning both the reward data and transition probabilities in a black-box environment setting, but fails to provide a viable scheme for modifying environment’s hyper-parameters to alter transition dynamics. 2 Preliminary and Problem Formulation In this section, we introduce three key entities in the RL data poisoning problem: the Markov decision process (MDP) that describes the environment, the RL agent, and the attacker that utilizes poisoning action. Their interactions are illustrated in Figure 1 . MDP Environment . We model the environment as a discounted MDP characterized by a tuple \langle\mathcal{S},\mathcal{A},r,P,\gamma\rangle , where \mathcal{S} denotes a finite state space, and \mathcal{A} denotes a finite action space. The reward r\in\mathbb{R}^{|\mathcal{S}||\mathcal{A}|} is specified as r=[r_{s,a}]_{s\in\mathcal{S},a\in\mathcal{A}} . The transition probability is defined as P:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1] . The discount factor \gamma\in(0,1] reflects our preference for future rewards against immediate ones. Figure 1: The interaction involves three entities: the agent, the environment and the attacker. The agent communicates its policy \pi to the environment, which then generates transition data based on the policy. However, the attacker intercepts