Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning

Roger Creus Castanyer, Johan Obando-Ceron, Pierre-Luc Bacon, Glen Berseth, Aaron Courville, Pablo Samuel Castro · 2025

Read Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learning by Roger Creus Castanyer, Johan Obando-Ceron, Pierre-Luc Bacon, Glen Berseth, Aaron Courville, Pablo Samuel Castro (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15544.txt.

Description

Roger Creus Castanyer and co-authors (2025) diagnose why bigger reinforcement learning agents often learn worse, and fix it with plain plumbing. Scaling a network's depth and width should add skill; in practice performance often crumbles. The authors' experiments point at two culprits acting together. The learning target keeps moving, since the agent's own behavior generates its data. And the gradients that carry learning signals degrade as they travel through deep networks, a pathology set by architecture choices.

Their remedies intervene directly on gradient flow: simple changes that keep gradients healthy through many layers. The fixes plug into standard algorithms without redesign. With them, agents hold strong performance as networks grow deep and wide, where before they sagged. The paper's claim is that much of the scaling problem was never about the learning rule's cleverness, but about whether the teaching signal survives the trip through the network.

Catalog record

CREATORRoger Creus Castanyer, Johan Obando-Ceron, Pierre-Luc Bacon, Glen Berseth, Aaron Courville, Pablo Samuel Castro
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15544

Browse the full library · About the Tower of Babel