1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities
Kevin Wang, Ishaan Javali, Michał Bortkiewicz, Tomasz Trzciński, Benjamin Eysenbach · 2025
Read 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities by Kevin Wang, Ishaan Javali, Michał Bortkiewicz, Tomasz Trzciński, Benjamin Eysenbach (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14858.txt.
Introduction
Kevin Wang and co-authors scale self-taught robot learning by stacking networks a thousand layers deep, in work from 2025. In this setup, no rewards or demos are given. The agent must explore on its own and learn to reach goals it is told. Most past work used shallow nets of two to five layers.
Depth up to over a thousand layers lifted scores on a contrast-based method by two to fifty times. Gains beat other goal-based rivals on fake walking and handling tasks. Depth did more than raise scores: the moves the agents learned changed in kind. The team shares code and a project page for the work.
Catalog record
| CREATOR | Kevin Wang, Ishaan Javali, Michał Bortkiewicz, Tomasz Trzciński, Benjamin Eysenbach |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-14858
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities Kevin Wang, Ishaan Javali, Michał Bortkiewicz, Tomasz Trzciński, Benjamin Eysenbach Abstract Scaling up self-supervised learning has driven breakthroughs in language and vision, yet comparable progress has remained elusive in reinforcement learning (RL). In this paper, we study building blocks for self-supervised RL that unlock substantial improvements in scalability, with network depth serving as a critical factor. Whereas most RL papers in recent years have relied on shallow architectures (around 2 – 5 layers), we demonstrate that increasing the depth up to 1024 layers can significantly boost performance. Our experiments are conducted in an unsupervised goal-conditioned setting, where no demonstrations or rewards are provided, so an agent must explore (from scratch) and learn how to maximize the likelihood of reaching commanded goals. Evaluated on simulated locomotion and manipulation tasks, our approach increases performance on the self-supervised contrastive RL algorithm by 2 × 2\times – 50 × 50\times , outperforming other goal-conditioned baselines. Increasing the model depth not only increases success rates but also qualitatively changes the behaviors learned. The project webpage and code can be found here: https://wang-kevin3290.github.io/scaling-crl/ . Figure 1: Scaling network depth yields performance gains across a suite of locomotion, navigation, and manipulation tasks, ranging from doubling performance to 50× improvements on Humanoid-based tasks. Notably, rather than scaling smoothly, performance often jumps at specific critical depths (e.g., 8 layers on Ant Big Maze, 64 on Humanoid U-Maze), which correspond to the emergence of qualitatively distinct policies (see Section 4 ). 1 Introduction While scaling model size has been an effective recipe in many areas of machine learning, its role and impact in reinforcement learning (RL) remain unclear. The typical model size for state-based RL tasks is between 2 to 5 layers ( Raffin et al., 2021 ; Huang et al., 2022 ) . In contrast, it is not uncommon to use very deep networks in other domain areas; Llama 3 ( Dubey et al., 2024 ) and Stable Diffusion 3 ( Esser et al., 2024 ) have hundreds of layers. In fields such as vision ( Radford et al., 2021 ; Zhai et al., 2021 ; Dehghani et al., 2023 ) and language ( Srivastava et al., 2023 ) , models often only acquire the ability to solve certain tasks once they are larger than a critical scale. In the RL setting, many researchers have searched for similar emergent phenomena ( Srivastava et al., 2023 ) , but these papers typically report only small marginal benefits and typically only on tasks where small models already achieve some degree of success ( Nauman et al., 2024b ; Lee et al., 2024 ; Farebrother et al., 2024 ) . A key open question in RL today is whether it is possible to achieve similar jumps in performance by scaling RL networks. At first glance, it makes sense why training very large RL networks should be difficult: the RL problem provides very few bits of feedback (e.g., only a sparse reward after a long sequence of observations), so the ratio of feedback to parameters is very small. The conventional wisdom ( LeCun, 2016 ) , reflected in many recent models ( Radford, 2018 ; Chen et al., 2020 ; Goyal et al., 2019 ) , has been that large AI systems must be trained primarily in a self-supervised fashion and that RL should only be used to finetune these models. Indeed, many of the recent breakthroughs in other fields have been primarily achieved with self-supervised methods, whether in computer vision ( Caron et al., 2021 ; Radford et al., 2021 ; Liu et al., 2024 ) , NLP ( Srivastava et al., 2023 ) , or multimodal learning ( Zong et al., 2024 ) . Thus, if we hope to scale reinforcement learning methods, self-supervision will likely be a key ingredient. In this paper, we will study building blocks for scaling reinforcement learning. Our first step is to rethink the conventional wisdom above: “reinforcement learning” and “self-supervised learning” are not diametric learning rules, but rather can be married together into self-supervised RL systems that explore and learn policies without reference to a reward function or demonstrations ( Eysenbach et al., 2021 ; Eysenbach et al., 2022 ; Lee et al., 2022 ) . In this work, we use one of the simplest self-supervised RL algorithms, contrastive RL (CRL) ( Eysenbach et al., 2022 ) . The second step is to recognize the importance of increasing data availability. We will do this by building on recent GPU-accelerated RL frameworks ( Makoviychuk et al., 2021 ; Rutherford et al., 2023 ; Rudin et al., 2022 ; Bortkiewicz et al., 2024 ) . The third step is to increase network depth, using networks that are up to 100\times deeper than those typically found in prior work. Stabilizing the training of such networks will require incorporating architectural techniques from prior work, including residual connections ( He et al., 2015 ) , layer normalization ( Ba et al., 2016 ) , and Swish activation ( Ramachandran et al., 2018 ) . Our experiments will also study the relative importance of batch size and network width. The primary contribution of this work is to show that a method that integrates these building blocks into a single RL approach exhibits strong scalability: • Empirical Scalability: We observe a significant performance increase, more than 20\times in half of the environments and outperforming other standard goal-conditioned baselines. These performance gains correspond to qualitatively distinct policies that emerge as the scale increases. • Scaling Depth in Network Architecture: While many prior RL works have primarily focused on increasing network width, they often report limited or even negative returns when expanding depth ( Lee et al., 2024 ; Nauman et al., 2024b ) . In contrast, our approach unlocks the ability to scale along the axis of depth, yielding performance improvements that surpass those from scaling width alone (see Sec. 4 ). • Empirical Analysis : We conduct an extensive analysis of the key components in our scaling approach, uncovering critical factors and offering new insights. We anticipate that future research may build on this foundation by uncovering additional building blocks. 2 Related Work Natural Language Processing (NLP) and Computer Vision (CV) have recently converged in adopting similar architectures (i.e. transformers) and shared learning paradigms (i.e self-supervised learning), which together have enabled transformative capabilities of large-scale models ( Vaswani et al., 2017 ; Srivastava et al., 2023 ; Zhai et al., 2021 ; Dehghani et al., 2023 ; Wei et al., 2022 ) . In contrast, achieving similar advancements in reinforcement learning (RL) remains challenging. Several studies have explored the obstacles to scaling large RL models, including parameter underutilization ( Obando-Ceron et al., 2024 ) , plasticity and capacity loss ( Lyle et al., 2024 ; Lyle et al., 2022 ) , data sparsity ( Andrychowicz et al., 2017 ; LeCun, 2016 ) , and training instabilities ( Ota et al., 2021 ; Henderson et al., 2018 ; Van Hasselt et al., 2018 ; Nauman et al., 2024a ) . As a result, current efforts to scale RL models are largely restricted to specific problem domains, such as imitation learning