On Corruption-Robustness in Performative Reinforcement Learning

Vasilis Pollatos, Debmalya Mandal, Goran Radanovic · 2025

Read On Corruption-Robustness in Performative Reinforcement Learning by Vasilis Pollatos, Debmalya Mandal, Goran Radanovic (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-05609.txt.

Introduction

Vasilis Pollatos, Debmalya Mandal, and Goran Radanovic study learning agents whose actions change the very world they learn from, in a paper from 2025. In performative reinforcement learning, the reward and the rules of motion depend on the agent's own policy. Past methods retrain again and again toward a stable policy.

The new work asks what happens when part of the data is corrupted by an adversary. The authors build a retraining method that uses robust estimates of the learning signal, designed for this problem's shape. They prove the method settles near a stable policy, with error growing gently as the corrupt share grows. Tests show that ignoring corruption hurts, while their guard keeps learning on track.

Catalog record

CREATORVasilis Pollatos, Debmalya Mandal, Goran Radanovic
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE137 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-05609

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On Corruption-Robustness in Performative Reinforcement Learning Vasilis Pollatos, Debmalya Mandal, Goran Radanovic Abstract In performative Reinforcement Learning (RL), an agent faces a policy-dependent environment: the reward and transition functions depend on the agent’s policy. Prior work on performative RL has studied the convergence of repeated retraining approaches to a performatively stable policy. In the finite sample regime, these approaches repeatedly solve for a saddle point of a convex-concave objective, which estimates the Lagrangian of a regularized version of the reinforcement learning problem. In this paper, we aim to extend such repeated retraining approaches, enabling them to operate under corrupted data. More specifically, we consider Huber’s ϵ \epsilon -contamination model, where an ϵ \epsilon fraction of data points is corrupted by arbitrary adversarial noise. We propose a repeated retraining approach based on convex-concave optimization under corrupted gradients and a novel problem-specific robust mean estimator for the gradients. We prove that our approach exhibits last-iterate convergence to an approximately stable policy, with the approximation error linear in ϵ \sqrt{\epsilon} . We experimentally demonstrate the importance of accounting for corruption in performative RL. 1 Archimedes/Athena RC, Greece, 2 University of Warwick, 3 Max Planck Institute for Software Systems 1 [email protected], 2 [email protected], 3 [email protected] Introduction In performative reinforcement learning ( Mandal, Triantafyllou, and Radanovic 2023 ; Rank et al. 2024 ) , the learner operates in a policy-dependent environment, where the reward and transition functions are influenced by the learner’s policy. A canonical example of such a setting is an RL system involving human users, such as RL-based recommender systems or chatbots: the RL policy affects user preferences, which in turn alters user engagement and behavior, thereby modifying the RL environment. For an offline learner, performativity represents a specific type of distribution shift. Similar to standard reinforcement learning (RL), where the learner’s policy influences the data it encounters due to the sequential decision-making process, performativity introduces an additional layer of complexity. The learner not only affects the observed data but also alters the underlying data generation process through its policy choices. This dual influence complicates the learning process, impacting the learner’s ability to generalize effectively. To alleviate this challenge prior work places sensitivity assumptions, formalized as Lipschitz conditions, which limit the extent the learner’s policy can influence the reward and transition functions of the underlying environment. These conditions enable convergence guarantees to a performatively stable policy during repeated retraining. This process involves optimizing a regularized RL objective after each deployment round to update the current policy. In the finite data regime, the retraining step solves for a saddle point of a convex-concave objective, which approximates the Lagrangian of the regularized reinforcement learning problem. In this paper, we additionally consider a third type of distribution shift, specifically induced by an adversary capable of corrupting training data. This scenario is particularly relevant in practical settings where performativity plays a role. For example, recommendation systems are vulnerable to Sybil attacks, while chatbots can be susceptible to poisoning poisoning attacks (e.g., see the Tay.ai incident ( Neff 2016 ) ). Therefore, it is crucial to explore strategies for achieving corruption-robustness in performative environments. Our goal is to extend algorithmic and convergence results from prior work on perfomrative RL by allowing an \epsilon fraction of data points in training data to be corrupted by adversarial noise—commonly referred to as the Huber \epsilon -contamination model . We recognize two main challenges: • First, this involves solving a convex-concave optimization problem where access to a given objective function is corrupted. For instance, gradient-based methods, which are commonly used for convex-concave optimization, rely on first-order gradient oracles to access the objective. However, existing guarantees for these methods typically assume that the outputs of gradient oracles are clean and not adversarially corrupted. • Second, generic robust estimators, such as those used for estimating gradients, do not account for the problem structure, often making them impractical in challenging high-dimensional settings. Contributions: Our work aims to resolve these challenges within the performative RL framework. Our contributions are as follow: • We propose a robust version of Optimistic Optimistic Follow the Regularized Leader (OFTRL) for optimizing convex-concave objectives under corruption and provide theoretical analysis of thereof. This analysis shows that the robust OFTRL achieves optimal convergence rates and information-theoretically optimal terminal error. These results are of an independent interest and complement the concurrent work on gradient-based algorithms for convex-concave optimization under corrupted gradients. See the related work section for more details. • We extend the existing algorithmic approach to performative reinforcement learning, making it robust against corruption. More specifically, we propose a novel repeated retraining approach based on robust OFTRL and a novel coordinate-wise robust mean estimator for estimating the gradients. The robust mean estimator is particularly suited for performative RL. We theoretically analyze our repeated retraining approach, showing that it exhibits last-iterate convergence to an approximately stable policy, with the approximation error that scales linearly with the square root of the corruption level \epsilon . • Using a simulation-based experimental testbed, we showcase the importance of accounting for corrupted gradients, and the efficacy of our approach. Related Work We recognize several lines of works related to this paper: performative prediction and RL , corruption-robust offline RL , and convex-concave optimization , and convex optimization under corrupted gradients . Performative Prediction and RL : Performative prediction models data distribution shifts due to the model deployment ( Perdomo et al. 2020 ) . Much prior work has studied convergence properties of algorithms to different solution concepts, including performative stability ( Perdomo et al. 2020 ; Mendler-Dünner et al. 2020 ) and performative optimality ( Izzo, Ying, and Zou 2021 ; Miller, Perdomo, and Zrnic 2021 ) . In recent years, other variants of the canonical setting have been proposed, including multi-player variants ( Narang et al. 2023 ; Piliouras and Yu 2023 ) , variants that consider more nuanced state-dependent distribution shifts ( Brown, Hod, and Kalemaj 2022 ; Li and Wai 2022 ) , or variants that introduce constraints ( Yan and Cao 2024 ) or bilevel optimization ( Lu 2023 ) . The closest to our setting is the work of ( Mandal, Triantafyllou, and Radanovic 2023 ) , who introduced the concept of performativity in reinforcement learning. The performative RL framework relates to Stackelberg stochastic games ( Letchford et al. 2012 ; Zhong et al. 2021 ) , in which a principal agent commits to a policy to which follower agents respond—making the principal’s effective environment performative. However, the performative RL framework abstracts away from game-theoretic considerations, as it does not model performativity through game-theoretic agents. ( Rank et al. 2024 ) extends the performative RL framework by considering gradual environment shifts, akin to those considered in the performative precision settings of ( Brown, Hod, and Kalemaj 2022 ; Li and Wai 2022 ) . Our paper contributes to the extensive literature on performative prediction ( Hardt and Mendler-Dünner 2023 ) by introducing corruption-robustness to performative RL. Corruption-Robust Offline RL : From a technical perspective, our results build on the analysis ( Mandal, Triantafyllou, and Radanovic 2023 ) , who adapted the minimax optimization problem ( Zhan et al. 2022 ) for the offline setting of performative RL. Hence, our work relates to

Read the complete text