Can In-Context Reinforcement Learning Recover From Reward Poisoning Attacks?
Paulius Sasnauskas, Yiğit Yalın, Goran Radanović · 2025
Read Can In-Context Reinforcement Learning Recover From Reward Poisoning Attacks? by Paulius Sasnauskas, Yiğit Yalın, Goran Radanović (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06891.txt.
Introduction
In-context reinforcement learning lets a trained model infer good actions from examples in its prompt, but an attacker who poisons the rewards in those examples may mislead it. Sasnauskas, Yalın, and Radanović study whether the Decision-Pretrained Transformer can withstand such reward poisoning.
They propose adversarial training. A population of attackers learns to poison rewards so as to cut the model's true reward, while the model learns to act well despite the poisoned data. In bandit tests the defended model beat standard algorithms, including robust ones built for contaminated rewards. It also held up against adaptive attackers and in richer Markov decision process settings, a promising sign for corruption-resistant learning.
Catalog record
| CREATOR | Paulius Sasnauskas, Yiğit Yalın, Goran Radanović |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06891
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can In-Context Reinforcement Learning Recover From Reward Poisoning Attacks? Paulius Sasnauskas, Yiğit Yalın, Goran Radanović Abstract We study the corruption-robustness of in-context reinforcement learning (ICRL), focusing on the Decision-Pretrained Transformer ( Lee et al. 2023 , DPT,) . To address the challenge of reward poisoning attacks targeting the DPT, we propose a novel adversarial training framework, called Adversarially Trained Decision-Pretrained Transformer (AT-DPT). Our method simultaneously trains an attacker to minimize the true reward of the DPT by poisoning environment rewards, and a DPT model to infer optimal actions from the poisoned data. We evaluate the effectiveness of our approach against standard bandit algorithms, including robust baselines designed to handle reward contamination. Our results show that the proposed method significantly outperforms these baselines in bandit settings, under a learned attacker. We additionally evaluate AT-DPT on an adaptive attacker, and observe similar results. Furthermore, we extend our evaluation to the MDP setting, confirming that the robustness observed in bandit scenarios generalizes to more complex environments. 1 Introduction Recent years have shown the impressive capabilities of transformer-based models on a range of tasks ( Vaswani et al. 2017 ; Raffel et al. 2020 ) . The community has been shifting from single-task learning, to multi-task learning, and even multi-domain learning ( Reed et al. 2022 ) . This has been made possible in part due to in-context learning, also called few-shot learning ( Brown et al. 2020 ) , which allows a model to adapt to new tasks simply by reading a handful of examples in the prompt, rather than requiring parameter updates or retraining. Recently, transformers and in-context learning have found growing use in decision-making tasks, particularly in reinforcement learning (RL), where interactions with the environment replace traditional text-based examples ( Chen et al. 2021a ; Xu et al. 2022 ; Laskin et al. 2022 ; Lee et al. 2023 ) . In this paper, we focus on the robustness of in-context RL to reward poisoning attacks – one of the major security threats for safe deployment of RL agents. Reward poisoning attacks have been extensively explored in recent RL literature ( Lin et al. 2017 ; Ma et al. 2019 ; Zhang et al. 2020b ; Wu et al. 2023 ; Nika et al. 2023 ) . This line of work predominantly focuses on the canonical RL setting, modeling reward poisoning attacks as an attacker that corrupts the reward of a learning agent during training. In contrast to test-time adversarial attacks, poisoning attacks influence the policy that the agent adopts at test time; i.e., they are training-time attacks. This is perhaps not surprising, given that this line of work typically focuses on Markov stationary policies, implying that the agent’s behavior is independent of the rewards at test time . However, an in-context RL agent can implement a learning algorithm in-context, using approaches such as Algorithm Distillation ( Laskin et al. 2022 ) or the Decision-Pretrained Transformer ( Lee et al. 2023 ) . In this case, contextual information encodes past interactions between the environment and the agent, including past rewards. By corrupting the agent’s rewards at test time, an adversary can still influence the agent’s behavior. Simply put, such test-time reward poisoning schemes attack the learning algorithm implemented in-context. In this work, we aim to develop a training protocol for in-context RL that enables models to be robust against test-time reward poisoning attacks targeting in-context learners. We focus on using the Decision-Pretrained Transformer (DPT) as a base approach. At a high level, the novel training protocol should implement a corruption-robust learning algorithm in-context. This differs from corruption-robust RL approaches typically studied in the literature ( Lin et al. 2017 ; Ma et al. 2019 ; Zhang et al. 2020b ; Sun et al. 2021 ; Wu et al. 2023 ; Nika et al. 2023 ) : in our setting, rewards are corrupted only at test time, meaning the corruption does not affect the training process of the agent’s policy. Our contributions are as follows. Framework. We introduce and formalize a novel attack modality predicated on reward poisoning. In this attack modality, the attacker influences the context that the agent’s policy is conditioned on by corrupting the agent’s rewards. More specifically, the attacker aims to minimize the agent’s return by perturbing the underlying reward function under a soft budget constraint encoded via a penalty term. Method. We combine in-context learning with adversarial training to develop an agent that is robust against reward poisoning. Specifically, we introduce the Adversarially Trained Decision-Pretrained Transformer (AT-DPT), which is trained by simultaneous optimization: an adversary tries to minimize the environment’s reward, while DPT learns to infer optimal actions from the corrupted data. An overview of the training procedure can be seen in Figure 1 . Experiments. We conduct a systematic evaluation of the proposed method, comparing its corruption-robustness capabilities to various baselines, including robust baselines designed to handle reward contamination, under various levels of poisoning. Our results show that AT-DPT can recover from a wide range of reward poisoning attacks and overall yields better performance under corruption than the baselines considered. We believe that the results presented in the work show potential of transformer-based policies in implementing algorithms that are robust against data contamination. Update \theta : minimize prediction loss AT-DPT \theta a^{*} s_{\text{query}} Dataset D^{\dagger} Environment \mathcal{M}_{i} Attacker \phi_{i} Dataset \bar{D} clean rewards corrupted rewards Update \phi_{i} with RL Figure 1 : The training procedure of AT-DPT. We use adversarial training to optimize the parameters \theta of a transformer model, which is our learning agent. In each round, we first collect data by deploying the agent in m environments. In environment i\in{0,\dots,m} , the agent observes rewards corrupted by an adversary defined by parameters \phi_{i} . We collect clean ( \bar{D} ) and corrupted ( D^{\dagger} ) datasets containing trajectories with clean and corrupted rewards, respectively. The agent is trained to predict an optimal action for a query state given a context sampled from a corrupted dataset. The adversary is trained to minimize the agent’s return under a soft budget constraint, expressed as a penalty term. During the test phase, the agent is deployed in a new corrupted environment. 2 Related work Adversarial ML. Adversarial ML is a line of work with the goal of understanding the effects of adversaries on models, also studying methods to defend against such adversaries. These adversaries have been extensively studied in computer vision, and more recently, several attacks have been proposed in RL. These include test-time attacks on observations ( Biggio et al. 2013 ; Szegedy et al. 2013 ; Goodfellow et al. 2015 ; Papernot et al. 2017 ) , (training-time) poisoning attacks ( Mei and Zhu 2015 ; Li et al. 2016 ) , and backdoor attacks Chen et al. 2017 ; Gu et al. 2017 ; Salem et al. 2022 . Closest to our work are poisoning attacks on RL ( Huang et al. 2017 ; Sun et al. 2021 ; Zhang et al. 2021 ) , which have considered different poisoning aims, including transitions ( Ma et al. 2019 ; Rakhsha et al. 2020 ) , rewards ( Lin et al. 2017 ; Zhang et al. 2020b ; Sun et al. 2020 ; Wu et al.