Confounding Robust Deep Reinforcement Learning: A Causal Approach
Mingxuan Li, Junzhe Zhang, Elias Bareinboim · 2025
Read Confounding Robust Deep Reinforcement Learning: A Causal Approach by Mingxuan Li, Junzhe Zhang, Elias Bareinboim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21110.txt.
Introduction
Mingxuan Li leads two co-authors on confounding robust deep reinforcement learning, a causal approach to off policy learning from biased data. Agents often must learn from past records where hidden factors skewed both actions and outcomes, and standard Q learning then chases biased rewards.
Building on the Deep Q Network, their algorithm seeks a safe policy that performs acceptably even under unobserved confounding of plausible strength. High dimensional domains are the target, where confounding cannot be ruled out in advance. Robust policies matter in health and recommendation settings, where logged data are plentiful but never randomized.
Catalog record
| CREATOR | Mingxuan Li, Junzhe Zhang, Elias Bareinboim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 91 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21110
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Confounding Robust Deep Reinforcement Learning: A Causal Approach Mingxuan Li, Junzhe Zhang, Elias Bareinboim Abstract A key task in Artificial Intelligence is learning effective policies for controlling agents in unknown environments to optimize performance measures. Off-policy learning methods, like Q-learning, allow learners to make optimal decisions based on past experiences. This paper studies off-policy learning from biased data in complex and high-dimensional domains where unobserved confounding cannot be ruled out a priori. Building on the well-celebrated Deep Q-Network (DQN), we propose a novel deep reinforcement learning algorithm robust to confounding biases in observed data. Specifically, our algorithm attempts to find a safe policy for the worst-case environment compatible with the observations. We apply our method to twelve confounded Atari games, and find that it consistently dominates the standard DQN in all games where the observed input to the behavioral and target policies mismatch and unobserved confounders exist. 1 Introduction Over the last decade, reinforcement learning (RL) has gained significant popularity for solving complex sequential decision-making problems, primarily due to its integration with deep learning techniques ( lecun2015deep ; schmidhuber2015deep ; goodfellow2016deep ) . This approach, known as deep reinforcement learning (deep RL), is particularly effective in high-dimensional state spaces ( mnihHumanlevelControlDeep2015a ; schulmanHighDimensionalContinuousControl2016a ; schulmanProximalPolicyOptimization2017 ; jumperHighlyAccurateProtein2021 ; akkaya2019solvingcube ; robocook2023Shi ) . Deep RL addresses the challenges faced by earlier RL methods by extracting various abstractions from data in complex domains with minimal prior knowledge. For example, a classic algorithm known as Deep Q-Network (DQN) algorithm can efficiently learn from visual inputs containing thousands of pixels ( mnihHumanlevelControlDeep2015a ) , enabling problem-solving capabilities comparable to humans in certain high-dimensional environments, such as Atari games for the first time. These achievements were followed by notable advancements in deep RL, including mastering the game of Go ( silverMasteringGameGo2017 ) , defeating world-class professionals at the game of poker ( moravvcik2017deepstack ) . Deep RL has also shown potentials in various real-world applications like robotics ( akkaya2019solvingcube ; robocook2023Shi ) , autonomous driving ( DBLP:journals/tits/KiranSTMSYP22autodriving ) and protein design ( jumperHighlyAccurateProtein2021 ) . These advancements eventually culminated in Sutton and Barto receiving the Turing Award in 2025 ( turing2024 ) . This paper attempts to leverage the capabilities and insights of DQN, while identifying an important assumption embedded in this algorithm and its variants that does not necessarily hold in the real world. Particularly, we notice that it is often implicitly assumed through the (PO)MDPs ( rlbook ) framework or explicitly enforced during the data-collection that no unmeasured confounder (NUC, ( robbins1985some ; crlsurvey ) ) affects the observed action and the subsequent outcomes. When the NUC does not hold, the effect of the target policy is generally not identifiable , i.e., the model assumptions are insufficient to uniquely determine the value function from the offline data ( pearl2009causality ; zhang2019near ) . On the other hand, partial identification is a line of methodologies that enable the derivation of informative bounds on target effects from confounded observations in non-identifiable settings ( Manski1989NonparametricBO ) . It has been studied under the rubrics of causal inference ( balke:pea97 ; zhang2022partial ) , econometrics ( imbens1997bayesian ; poirier1998revising ; romano2008inference ; stoye2009more ; bugni2010bootstrap ; todem2010global ; moon2012bayesian ) , and dynamical systems ( bajari2007estimating ; norets2014semiparametric ; dickstein2018exporters ; morales2019extended ; berry2023instrumental ) . More recently, researchers have been using partial identification methods to obtain reliable off-policy evaluation in reinforcement learning ( kallus2018confounding ; zhang2019near ; kallus2020confounding ; namkoong2020off ; khan2023off ; bruns2023robust ; kausik2024offline ; zhang2025eligibility ; li2025confoundedshaping ; DBLP:conf/aaai/JoshiZB24safepolicycausal ) . Despite these achievements, significant challenges still exist in applying partial identification for policy learning in complex and high-dimensional domains, including images and videos. We refer readers to Appendix A for a more detailed survey on partial identification and deep reinforcement learning. This paper aims to address these challenges by investigating deep reinforcement learning algorithms from offline data over complex and high-dimensional domains, where the presence of unmeasured confounders could not be assumed away a priori . More specifically, our contributions are summarized as follows. (1) We introduce a novel DQN algorithm, which we call Causal DQN, capable of learning robust abstractions from confounded data over complex and high-dimensional domains with minimal prior knowledge. (2) We empirically demonstrate that our method significantly improves robustness and generalization under confounded observations and outperforms various DQN baselines across twelve popular Atari games. Due to space constraints, details of the experiment setup and additional experiments are provided in Appendices D and E . Videos of gameplay are included in the supplemental. Notations. We will consistently use capital letters ( V ) to denote random variables, lowercase letters ( v ) for their values, and cursive \mathcal{V} to denote the their domains. We use bold capital letters ( \boldsymbol{V} ) to denote a set of random variables and let |\boldsymbol{V}| denote its cardinality of set \boldsymbol{V} . Finally, \boldsymbol{1}_{\boldsymbol{Z}=\boldsymbol{z}} is an indicator function that returns 1 if event \boldsymbol{Z}=\boldsymbol{z} holds true; otherwise, it returns 0 . 2 Challenges Due to Unobserved Confounders We will focus on a sequential decision-making problem in the Markov Decision Process (MDP, ( putermanMarkovDecisionProcesses1994book ) ) where the agent intervenes on a sequence of actions to optimize subsequent rewards. Standard MDP models focus on the perspective of learners who could actively intervene in the environment. Consequently, confounding is generally assumed away a priori. On the other hand, when considering off-policy data collected from passive observations, the learner does not necessarily have the liberty to control how the behavioral policy generates the data, giving rise to unobserved confounders in decision-making tasks ( kallus2018confounding ; zhang2020causal ; kumor2021causal ; guo2022provably ; ruan2024causal ) . In this paper, we will consider a generalized family of confounded MDPs ( zhang2022can ; bennett2021off ; kallus2020confounding ; zhang2025eligibility ; li2025confoundedshaping ) explicitly modeling the presence of unobserved confounders in the off-policy data generation. Definition 2.1 . A Confounded Markov Decision Process (CMDP) \mathcal{M} is a tuple of \langle\mathcal{S},\mathcal{X},\mathcal{Y},\mathcal{U},\mathcal{F},P\rangle where (1) \mathcal{S},\mathcal{X},\mathcal{Y} are, respectively, the space of observed states, actions, and rewards; (2) \mathcal{U} is the space of unobserved exogenous noise; (3) \mathcal{F} is a set consisting of the transition function f_{S}:\mathcal{S}\times\mathcal{X}\times\mathcal{U}\mapsto\mathcal{S} , behavioral policy f_{X}:\mathcal{S}\times\mathcal{U}\mapsto\mathcal{X} , and reward function f_{Y}:\mathcal{S}\times\mathcal{X}\times\mathcal{U}\mapsto\mathcal{Y} ; (4) P is an exogenous distribution over the domain \mathcal{U} . S 1 X 1 Y 1 S 2 X 2 Y 2 S 3 X 3 Y 3 Figure 1 : Causal diagram representing the data-generating mechanisms in a Confounded Markov Decision Process. Throughout this paper, we will consistently assume the action domain \mathcal{X} to be discrete and finite, while the state domain \mathcal{S} could be complex and continuous; the reward domain \mathcal{Y} is bounded in a real interval [a,b]\subset\mathbb{R} . Consider a demonstrator agent interacting with a CMDP \mathcal{M} , generating the off-policy data. For every time step t=1,\dots,T , the environment first draws an exogenous noise U_{t} from the distribution P(\mathcal{U}) ; the demonstrator then performs an action X_{t}\leftarrow f_{X}(S_{t},U_{t}) , receives a subsequent reward Y_{t}\leftarrow r_{t}(S_{t},X_{t},U_{t}) , and moves to the next state S_{t+1}\leftarrow f_{S}(S_{t},X_{t},U_{t}) . The observed trajectories of the demonstrator (from the learner’s perspective) are summarized as the observational distribution P(\bar{\boldsymbol{X}}_{1:T},\bar{\boldsymbol{S}}_{1:T},\bar{\boldsymbol{Y}}_{1:T}) , i.e., \displaystyle P(\bar{\boldsymbol{x}}_{1:T},\bar{\boldsymbol{s}}_{1:T},\bar{\boldsymbol{y}}_{1:T})=P(s_{1})\prod_{t=1}^{T}\bigg(\int_{\mathcal{U}}\boldsymbol{1}_{s_{t+1}=f_{S}(s_{t},x_{t},u_{t})}\boldsymbol{1}_{x_{t}=f_{X}(s_{t},u_{t})}\boldsymbol{1}_{y_{h}=f_{Y}(s_{t},x_{t},u_{t})}P(u_{t})\bigg) Figure 1 shows the causal diagram \mathcal{G}