Statistical Reinforcement Learning in the Real World: A Survey of Challenges and Future Directions
Asim H. Gazi, Yongyi Guo, Daiqi Gao, Ziping Xu, Kelly W. Zhang, Susan A. Murphy, [1em] Department of Computer Science, Harvard University, [0.5em] Equal contribution · 2026
Read Statistical Reinforcement Learning in the Real World: A Survey of Challenges and Future Directions by Asim H. Gazi, Yongyi Guo, Daiqi Gao, Ziping Xu, Kelly W. Zhang, Susan A. Murphy, [1em] Department of Computer Science, Harvard University, [0.5em] Equal contribution (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15353.txt.
Introduction
Asim H. Gazi, Yongyi Guo, and co-authors survey why reinforcement learning still struggles outside the lab. Their 2026 arXiv paper deals with statistical reinforcement learning in the real world. The field has won in gaming, robotics, online ads, public health, and language tasks.
Yet a wide gap stands between research results and deployed systems. Two recurring challenges underlie much of that gap. The survey names the main obstacles to real use. It then sets out future directions that could one day close the distance between journal papers and daily real-world practice.
Catalog record
| CREATOR | Asim H. Gazi, Yongyi Guo, Daiqi Gao, Ziping Xu, Kelly W. Zhang, Susan A. Murphy, [1em] Department of Computer Science, Harvard University, [0.5em] Equal contribution |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-15353
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Statistical Reinforcement Learning in the Real World: A Survey of Challenges and Future Directions Asim H. Gazi, Yongyi Guo, Daiqi Gao, Ziping Xu, Kelly W. Zhang, Susan A. Murphy, [1em] Department of Computer Science, Harvard University, [0.5em] Equal contribution Abstract Reinforcement learning (RL) has achieved remarkable success in real-world decision-making across diverse domains, including gaming, robotics, online advertising, public health, and natural language processing. Despite these advances, a substantial gap remains between RL research and its deployment in many practical settings. Two recurring challenges often underlie this gap. First, many settings offer limited opportunity for the agent to interact extensively with the target environment due to practical constraints. Second, many target environments often undergo substantial changes, requiring redesign and redeployment of RL systems (e.g., advancements in science and technology that change the landscape of healthcare delivery). Addressing these challenges and bridging the gap between basic research and application requires theory and methodology that directly inform the design, implementation, and continual improvement of RL systems in real-world settings. In this paper, we frame the application of RL in practice as a three-component process: (i) online learning and optimization during deployment, (ii) post- or between-deployment offline analyses, and (iii) repeated cycles of deployment and redeployment to continually improve the RL system. We provide a narrative review of recent advances in statistical RL that address these components, including methods for maximizing data utility for between-deployment inference, enhancing sample efficiency for online learning within-deployment, and designing sequences of deployments for continual improvement. We also outline future research directions in statistical RL that are use-inspired—aiming for impactful application of RL in practice. Keywords: Reinforcement learning, adaptive experiments, adaptive interventions, online learning, statistical inference, sequential deployments 1 Introduction Reinforcement learning (RL) has received considerable interest in recent years due to superhuman or state-of-the-art performance in areas such as gaming, autonomous driving, and robotics ( 131 ; 172 ; 201 ; 184 ; 92 ) . One of the factors behind these real-world successes is the proliferation of deep learning techniques that enable flexible learning from big data. Common characteristics of applications where deep RL has success include (1) areas in which the RL algorithm can interact extensively with the target environment or a high-fidelity simulator, and (2) minimal changes in the environment’s key variables and dynamics such that re-learning via online RL and/or continual redesign of the RL algorithm is often not required or only infrequently required for successful decision-making by the algorithm in future deployments. For example, a recent high-profile example of RL success in the real world is the use of deep RL to defeat champion human racers in Gran Turismo ( 201 ) , a highly realistic automobile racing simulator. In this work, the authors train a deep RL agent entirely offline in simulation—using population-based training beyond pure self-play to improve robustness to diverse opponent behaviors—and subsequently deploy the learned policy as a fixed controller without online learning or algorithm updates. This success relies in part on the availability of a high-fidelity simulator, which permits extensive interaction with the target environment and large-scale exploration without real-world risks such as automobile crashes. Moreover, although exogenous factors such as opponent behavior may vary across races, the core components of the decision problem (e.g., state representation, action space, reward definition, and governing dynamics) do not change substantially across deployments. As a result, the learned policy does not require continual re-learning or algorithmic modification to remain effective in future use. When the two aforementioned characteristics do not apply, RL can be far more challenging to apply in the real world ( 52 ) 1 1 1 By “real-world,” we refer to RL applications within “Levels 1-5” of the taxonomy of real-world successes recently proposed by 184 . In contrast to Level 0, which consists of RL algorithms only validated in simulation environments, Level 1 consists of RL solutions that have been validated under limited lab conditions, Level 2 consists of RL solutions validated under diverse lab conditions, and so on (Level 5 consists of RL algorithms deployed in real-world products) . In our review of the literature, we find that this is the case especially when RL algorithms interact with humans. This is driven by two challenges : (1) the inability to interact extensively with the target environment or a high-fidelity simulator to obtain a large amount of data and (2) the presence of significant changes in the environment that require re-optimizing the decision-making policy by the RL algorithm during next deployment and/or updating the RL algorithm for use the next deployment. An example where RL algorithms interact with humans, have had some recent real-world success (e.g., digital interventions deployed in clinical trials), but still require substantial research and development is the use of RL to personalize just-in-time adaptive interventions (JITAIs) ( 63 ) . JITAIs make sequences of decisions during everyday life to support an individual with their health. RL has gained traction as an online learning and optimization approach for personalizing JITAIs. RL algorithms can be used to select actions about which intervention option to deliver (e.g., whether or not to deliver a push notification to an individual’s phone) based on state information sensed and surveyed from the individual (e.g., heart rate). The RL algorithm then observes how the individual responds and, using this data, learns online to optimize future decisions accordingly. Although RL has shown promise in recent real-world digital intervention clinical trials ( 7 ; 112 ; 191 ; 71 ; 114 ) , challenges remain that have limited further translation. Related to challenges (1) and (2) above, every individual and deployment is different, so even if extensive previous data exist, the mismatch between individuals in a new deployment and individuals in a previous deployment may be significant. Society and digital health technology also change rapidly over the span of only a few years, resulting in changing relationships between action, reward, or state variables, as well as changes in the variables themselves ( 4 ; 65 ) . Online learning can help account for these mismatches, but exploration is limited by the number of interactions with an individual—complete disengagement from an intervention can occur due to poor delivery (e.g., removal of the JITAI application) ( 136 ) . Figure 1: Reinforcement learning (RL) in practice, framed as a three-component process. The process involves online learning and optimization within each deployment, offline learning and inference between deployments, and a continual deployment-redeployment feedback process for continual improvement. For RL to make a substantial impact in the aforementioned real-world applications, challenges (1) and (2) must be addressed. Fig. 1 provides a forward-looking framework of implementing RL in real-world systems, a three-component process involving online autonomous learning and optimization while a RL algorithm is deployed, offline learning and statistical inference between deployments of a RL system, and a continuing deployment-redeployment process for continual improvement over the course of a RL system’s lifetime. This continuing deployment-redeployment process is related to work on offline-to-online RL ( 72 ; 21 ; 95 ) , where policies learned offline are fine tuned via online RL, but this framework is broader and includes not only offline-to-online RL (i.e., warm starting the next deployment), but also online-to-offline knowledge transfer (i.e., creating generalizable knowledge