The Formalism-Implementation Gap in Reinforcement Learning Research

Pablo Samuel Castro · 2025

Read The Formalism-Implementation Gap in Reinforcement Learning Research by Pablo Samuel Castro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16175.txt.

Introduction

Pablo Samuel Castro argues that reinforcement learning research has lost its balance. In this field, AI agents learn by trial and reward, and recent fame came from agents beating humans at set tasks. Castro says the rush to top scoreboards has crowded out work on why these agents learn as they do.

He makes two pleas. Study the science of how the learning works. And state more clearly how test games map onto the math behind them. His main case is the Arcade Learning Environment, a suite of old Atari games long called played out. Castro shows this suite can still teach deep lessons if used with care. Better basic science, he argues, is what will carry these methods into real-world use.

Catalog record

CREATORPablo Samuel Castro
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-16175

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Formalism-Implementation Gap in Reinforcement Learning Research Pablo Samuel Castro Abstract The last decade has seen an upswing in interest and adoption of reinforcement learning (RL) techniques, in large part due to its demonstrated capabilities at performing certain tasks at “super-human levels”. This has incentivized the community to prioritize research that demonstrates RL agent performance , often at the expense of research aimed at understanding their learning dynamics. Performance-focused research runs the risk of overfitting on academic benchmarks – thereby rendering them less useful – which can make it difficult to transfer proposed techniques to novel problems. Further, it implicitly diminishes work that does not push the performance-frontier, but aims at improving our understanding of these techniques. This paper argues two points: (i) RL research should stop focusing solely on demonstrating agent capabilities, and focus more on advancing the science and understanding of reinforcement learning; and (ii) we need to be more precise on how our benchmarks map to the underlying mathematical formalisms. We use the popular Arcade Learning Environment ( Bellemare et al. 2013 , ALE;) as an example of a benchmark that, despite being increasingly considered “saturated”, can be effectively used for developing this understanding, and facilitating the deployment of RL techniques in impactful real-world problems. 1 Introduction Reinforcement learning (RL) has been a subject of research for at least 40 years, but it is only in the last decade where we have experienced a dramatic increase in its adoption beyond academic pursuits. The pivot point can likely be attributed to Mnih et al. 2015 , who introduced DQN (standing for D eep Q - N etwork) which combined RL with deep neural networks and could play arcade games at super-human levels. The authors utilized the Arcade Learning Environment (ALE), which was introduced as “a platform and methodology for evaluating the development of general, domain-independent AI technology”, presenting “significant research challenges for RL” ( Bellemare et al. 2013 ) . For a number of years the ALE served as a useful benchmark for evaluating how far we could push the performance of RL agents when trained for 200 million screen frames (around 38 hours of game play). Later, Łukasz Kaiser et al. 2020 introduced the 100k benchmark (just under 2 hours of game play) to test the sample efficiency of RL agents. Even in the low-data regime of the 100k benchmark, there are already RL agents capable of achieving super-human performance ( Schwarzer et al. 2023 ) . The fact that performance gains in the ALE are increasingly difficult to obtain have led to a general sense that the ALE has become a saturated and uninteresting benchmark. As such, there is an implicit incentive for researchers to focus on other evaluation benchmarks where gains are less difficult to demonstrate. Unfortunately most of these benchmarks, including the ALE, lack a precise mapping to the underlying mathematical framework describing RL problems. When viewed simply as a platform for evaluating agent performance , it is hard to argue against this somewhat defeatist perspective. However, it is worth recalling that the objective of most RL research is not to develop super-human video game players; rather, our hope is that the techniques and methods we develop may be useful for addressing impactful and challenging real-world problems. To do so effectively would require a robust understanding of how to transfer techniques developed on one set of environments to new, and likely very different, ones. Following the insight-oriented exploratory research advocated by Herrmann et al. 2024 and the scientific testing suggested by Jordan et al. 2024 , RL research should focus on developing this understanding. Transferring an algorithm evaluated on one benchmark to another environment can be rather complex. As Sutton and Barto 1998 writes: “Applications of reinforcement learning are still far from routine and typically require as much art as science. Making applications easier and more straightforward is one of the goals of current research in reinforcement learning”; a statement that is still true almost thirty years later. Yet, there are a growing number of examples of RL being used successfully in real-world applications (see appendix B for a list of some examples); however, most (if not all) of these have required employing a team of RL experts to achieve success. This lack of transfer may be due to the fact that as a community we have over-indexed on environment-centric concepts, rather than agents (the “environment spotlight dogma” ( Abel et al. 2024 ) ). A complementary perspective is that we have focused on demonstrating agent capabilities (i.e. how well it performs against a baseline) as opposed to how and why an agent performs the way it does. Successful transfer would require properly understanding both (1) how algorithmic specifications (defined in abstract, mathematical terms) transfer to environment implementations (which include a number of non-trivial implementation design choices), and (2) the sensitivity of an agent to varying hyper-parameters and environmental properties (see fig. 1 ). This paper argues that, in order to develop transferable RL techniques, we need to be more formally explicit with benchmark specifications, and we need to stop focusing solely on performance gains. This shift in focus can help facilitate the use of reinforcement learning in more challenging and impactful real-world problems. Figure 1 : RL Algorithms are typically defined using the mathematical formalism of MDPs, but implemented (somewhat) independently with code, and evaluated on standard benchmarks such as the ALE. If we want to maximize the transferability of our algorithms, we need to be explicit about the MDP-benchmark and algorithm-implementation mappings . 2 The formalism-implementation gap RL algorithms are typically presented abstractly using the mathematical formalism of Markov decision processes, which enables their potential application to a wide variety of problems. These algorithms are then compared to baselines on a standard benchmark such as the ALE with the implication that stronger performance would transfer to other problems of interest. As fig. 1 illustrates, this requires explicit MDP-benchmark and algorithm-implementation mappings. Although the latter mapping is discussed more frequently, implementations necessarily involve a set of hyper-parameters to tune, which are not always broadly explored. As a number of works have demonstrated, an algorithm’s performance is often mostly due to hyper-parameter settings rather than its mathematical properties. For instance, Henderson et al. 2018 demonstrated how sensitive algorithms can be to choices of network architectures, reward scaling, and implementations; while Łukasz Kaiser et al. 2020 claimed their method (SimPLe) outperformed Rainbow ( Hessel et al. 2018 ) in the Atari 100k benchmark, Van Hasselt et al. 2019 demonstrated that with proper tuning, Rainbow was significantly stronger; Obando-Ceron and Castro 2021 showed significant improvements in DQN by simply switching the optimizer from RMSProp (as used by Mnih et al. 2015 ) to Adam ( Kingma and Ba 2015 ) ; and Ceron et al. 2024 performed an exhaustive evaluation of various hyper-parameters, convincingly demonstrating how much they affect performance and algorithmic rankings. 2.1 Markov decision processes A Markov decision process (MDP) is a tuple \mathcal{M}:=\langle\mathcal{X},\mathcal{A},\rho_{0},\mathcal{R},\mathcal{P}\rangle , where \mathcal{X} and \mathcal{A} denote the state and action spaces, respectively; \rho_{0}\in\Delta(\mathcal{X}) (for any set C , \Delta(C) denotes a distribution over C ) is a distribution over start states; \mathcal{R}:\mathcal{X}\times\mathcal{A}\rightarrow\mathbb{R} is a reward (or cost) function; and \mathcal{P}:\mathcal{X}\times\mathcal{A}\rightarrow\Delta(\mathcal{X})

Read the complete text