Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning

Ghada Sokar, Pablo Samuel Castro · 2025

Read Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning by Ghada Sokar, Pablo Samuel Castro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17749.txt.

Introduction

Ghada Sokar and Pablo Samuel Castro wrote a paper in two thousand twenty-five on why deep reinforcement learning agents trained on raw pixels fail to scale. Larger networks often perform worse in such settings, and the cause was unclear. Earlier fixes patched the issue indirectly through new designs or training tricks.

The authors trace the problem to one junction: where the image encoder, a stack of layers that read the pixels, hands its output to the dense layers that decide actions. They call this junction the bottleneck. Their analysis shows earlier remedies were all easing pressure at that same point. They propose global average pooling as a simple, direct fix for the bottleneck. The method avoids the complexity of the earlier workarounds.

Catalog record

CREATORGhada Sokar, Pablo Samuel Castro
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17749

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning Ghada Sokar, Pablo Samuel Castro Abstract Scaling deep reinforcement learning in pixel-based environments presents a significant challenge, often resulting in diminished performance. While recent works have proposed algorithmic and architectural approaches to address this, the underlying cause of the performance drop remains unclear. In this paper, we identify the connection between the output of the encoder (a stack of convolutional layers) and the ensuing dense layers as the main underlying factor limiting scaling capabilities; we denote this connection as the bottleneck , and we demonstrate that previous approaches implicitly target this bottleneck. As a result of our analyses, we present Global Average Pooling (GAP) as a simple yet effective way of targeting the bottleneck, thereby avoiding the complexity of earlier approaches. 1 Introduction Reinforcement Learning (RL) is widely considered one of the most effective approaches for complex sequential decision-making problems ( Mnih et al., 2015 ; Vinyals et al., 2019 ; Bellemare et al., 2020 ; Degrave et al., 2022 ; Wurman et al., 2022 ) , in particular when combined with deep neural networks (typically referred to as deep RL). In contrast to the so-called “scaling laws” observed in supervised learning (where larger networks typically result in improved performance) ( Kaplan et al., 2020 ) , it is difficult to scale RL networks without sacrificing performance. There has been a recent line of work aimed at developing techniques for effectively scaling value-based networks, such as via the use of mixtures-of-experts ( Obando Ceron* et al., 2024 ) , network pruning ( Obando Ceron et al., 2024 ) , tokenization ( Sokar et al., 2025 ) , and regularization ( Nauman et al., 2024 ) . Most of these techniques tend to focus on structural modifications to standard deep RL networks by leveraging sparse-network training techniques from the supervised learning literature. Obando Ceron* et al. (2024) first demonstrated that naïvely scaling the penultimate (dense) layer in an RL network results in decreased performance, and proposed the use of soft mixtures-of-experts ( Puigcerver et al., 2024 , SoftMoEs;) to enable improved performance from this form of scaling. Sokar et al. (2025) argued that the gains from SoftMoEs were mostly due to the use of tokenization. Relatedly, Obando Ceron et al. (2024) demonstrated that naïvely scaling the convolutional layers hurts performance, and showed that incremental parameter pruning yields gains that grow with the size of the original, unpruned, network. While effective, all these methods are non-trivial to implement and can result in increased computational costs. One unifying aspect of the aforementioned works is that they tend to be most effective on networks that process pixel inputs, such as when training on the Arcade Learning Environment (ALE) ( Bellemare et al., 2013 ) . These networks are typically divided into an encoder \phi consisting of a series of convolutional layers, followed by a series of dense layers \psi ; thus, for an input x , the network output is given by \psi(\phi(x)) . Obando Ceron* et al. (2024) and Sokar et al. (2025) scaled the first layer of \psi while Obando Ceron et al. (2024) scaled all layers in \phi . It is worth highlighting that \psi\circ\phi is, in practice, a set of weights connecting the output of \phi(x) with the input layer of \psi . In this work, we argue that the underlying cause behind the effectiveness of the aforementioned methods is that they result in a bottleneck between \phi(x) and \psi (see Figure 1 ). As we will argue, performance gains are mostly due to a well-structured bottleneck, rather than the recently proposed architectural modifications to \psi . This insight suggests that simpler architectural interventions may be just as effective, which we demonstrate by using Global Average Pooling (GAP) as an effective mechanism for enabling improved performance from scaling. Our contributions can be summarized as follows: (i) We investigate the challenges leading to the performance degradation of scaled RL networks; (ii) we study the underlying reasons behind the success of existing architectural approaches in scaling; and (iii) we present pooling as a faster, simpler method yielding superior performance. We begin in Section 2 by providing the necessary background on reinforcement learning, detailing the architecture employed in pixel-based deep RL, and outlining architectural modifications proposed by recent methods for addressing scaling challenges. Section 3.1 then describes our experimental setup. Our analyses investigating the difficulties of scaling RL networks and how existing methods attempt to address them are presented in Sections 3.2 and 3.3 , respectively. Section 4 is dedicated to our results and analysis on RL networks with GAP. Finally, Section 5 covers the related work, followed by a conclusion and discussion in Section 6 . Figure 1 : Illustration of the bottleneck in pixel-based networks. Standard dense networks ( Baseline ) connect all \phi outputs with \psi , resulting in H\times W\times C\times dim(\psi) parameters (scaled down when using pruning , shown in red). SoftMoE-1 converts \phi ’s outputs into H\times W tokens of dimension C ; the sharing of learned parameters across tokens results in a bottleneck with C\times dim(\psi) parameters. GAP performs average pooling across H\times W spatial dimensions, resulting in C feature maps and C\times dim(\psi) parameters in the bottleneck. 2 Preliminaries 2.1 Reinforcement learning Reinforcement learning involves an agent moving through a series of states x\in\mathcal{X} by selecting an action a\in\mathcal{A} at discrete timesteps. After selecting action a_{t} from state x_{t} , the agent will receive a reward r_{t}(x_{t},a_{t}) , and its goal is to maximize the discounted sum of cumulative rewards \sum_{t=0}^{\infty}\gamma^{t}r_{t} , where \gamma\in[0,1) by finding an optimal policy \pi:\mathcal{X}\rightarrow\Delta(\mathcal{A}) which quantifies the agent’s behavior at each state. Value-based methods ( Sutton and Barto, 1998 ) maintain estimates of the value of selecting action a from state x and following \pi afterwards: Q(x,a):=\mathbb{E}\left[\sum_{t=0}^{\infty}\left[\gamma^{t}r_{t}(x_{t},a_{t})|x_{0}=x,a_{0}=a,a_{t}\sim\pi(x_{t})\right]\right] , where \pi is induced from Q , for instance with the use of softmax: \pi(x)(a):=\frac{e^{Q(x,a)}}{\sum_{a^{\prime}\in\mathcal{A}}e^{Q(x,a^{\prime})}} . Mnih et al. (2015) demonstrated deep neural networks can be very effective at approximating Q -values, even for complex domains such as Atari games ( Bellemare et al., 2013 ) ; their network has served as the backbone for most deep RL networks. Later, Espeholt et al. (2018) proposed using ResNet based architecture which demonstrates significant performance improvements over the original convolutional neural network (CNN) architecture. For pixel-based environments, this family of networks consists of a set of convolutional layers ( Fukushima, 1980 ) , which we will collectively refer to as \phi (and often referred to as the encoder or representation ), followed by a set of dense layers, which we will collectively refer to as \psi . Thus given an input x , the network approximates the Q -values as \tilde{Q}(x,\cdot)=\psi(\phi(x)) . The output of \phi is a 3-dimensional tensor H\times W\times C , where H is height, W is width, and C is the number of feature maps (channels); this output is typically flattened before being fed to \psi . Thus, the number of parameters for the functional composition \psi\circ\phi is equal to H\times W\times C\times dim(\psi) , where dim(\psi) is the dimensionality of the first dense layer in \psi . We refer to connection between \phi and \psi as

Read the complete text