Evolutionary Discovery of Reinforcement Learning Algorithms via Large Language Models Conference: ; ;

Alkis Sygkounas, Amy Loutfi, Andreas Persson · 2026

Read Evolutionary Discovery of Reinforcement Learning Algorithms via Large Language Models Conference: ; ; by Alkis Sygkounas, Amy Loutfi, Andreas Persson (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28416.txt.

Introduction

Sygkounas, Loutfi, and Persson evolve new reinforcement learning algorithms by searching over the update rules themselves. Usual algorithms are hand-designed and fixed. Their search, built on the REvolve system, uses language models to vary executable training rules. Common crutches like actor-critic forms and value bootstrapping are banned, to force fresh ideas. A later pass tunes each rule's scalar settings. Full training runs on Gymnasium tasks show the found algorithms rival SAC, PPO, and DQN.

Catalog record

CREATORAlkis Sygkounas, Amy Loutfi, Andreas Persson
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70207
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.28416

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evolutionary Discovery of Reinforcement Learning Algorithms via Large Language Models Conference: ; ; Alkis Sygkounas, Amy Loutfi, Andreas Persson Abstract . Reinforcement learning algorithms are defined by their learning update rules, which are typically hand-designed and fixed. We present an evolutionary framework for discovering reinforcement learning algorithms by searching directly over executable update rules that implement complete training procedures. The approach builds on REvolve, an evolutionary system that uses large language models as generative variation operators, and extends it from reward-function discovery to algorithm discovery. To promote the emergence of nonstandard learning rules, the search excludes canonical mechanisms such as actor–critic structures, temporal-difference losses, and value bootstrapping. Because reinforcement learning algorithms are highly sensitive to internal scalar parameters, we introduce a post-evolution refinement stage in which a large language model proposes feasible hyperparameter ranges for each evolved update rule. Evaluated end-to-end by full training runs on multiple Gymnasium benchmarks, the discovered algorithms achieve competitive performance relative to established baselines, including SAC, PPO, DQN, and A2C. 1. Introduction In reinforcement learning, the update rule determines how experience is transformed into parameter updates, which directly shape the learning behavior. Although automated discovery has been applied to some aspects of reinforcement learning, such as architecture design ( 36 ; 27 ; 31 ) , hyperparameter tuning ( 26 ; 35 ; 5 ) , and training-signal design ( 10 ; 37 ) , the update rule typically remains fixed. A small number of works have attempted to further elevate the discovery by exploring the learning of update rules intrinsically, either by optimizing differentiable update functions ( 22 ) , or by evolving structured loss expressions ( 4 ) . However, these methods operate over restricted, parameterized representations of the learning rule, such as differentiable update networks or symbolic loss forms, and therefore cannot generate fundamentally different update rules expressed as executable training logic. As a result, the update rule remains one of the core components of reinforcement learning that has not yet been the subject of fully automated design. Searching over reinforcement-learning update rules is incompatible with gradient-based or other locally guided optimization because the search space consists of discrete program structures implementing full training procedures, rather than continuous parameters or differentiable update modules. Even small changes to the update logic can induce qualitatively different learning dynamics, and reinforcement-learning algorithms are known to be highly sensitive to implementation details ( 12 ; 6 ) . Hyperparameters that stabilize one update rule often fail for another ( 25 ) , preventing effective local refinement through small perturbations. As a result, each update rule must be treated as a distinct algorithm whose quality can only be assessed through complete training runs. Under these conditions, population-based evolutionary methods are a natural fit, since they compare candidate algorithms using end-to-end performance without requiring gradients or smooth objective structure ( 13 ; 15 ; 4 ) . Evolutionary algorithms and genetic programming have been applied to a wide range of machine-learning tasks, including neural network evolution, hyperparameter optimization, controller synthesis, and symbolic policy search ( 8 ; 13 ; 15 ) . A key practical requirement in these settings is the ability to generate offspring that are both syntactically valid and executable, enabling evolutionary operators to explore the search space without producing predominantly invalid variants ( 4 ) . Reinforcement-learning update rules, however, violate this requirement because they are implemented as tightly coupled training procedures rather than isolated objective functions. As a result, even small structural code changes can break dependencies between components or invalidate required data flows, leading to failed or undefined training behavior ( 12 ; 6 ; 25 ) . Consequently, unguided mutation and crossover are unlikely to reliably produce functional update rules, underscoring the need for a guided mechanism that proposes coherent, executable modifications to complete training logic. Recent work suggests that large language models (LLMs) can serve as such proposal mechanisms within evolutionary pipelines, enabling structured exploration of complex design spaces. In reinforcement learning, this approach has been used to optimize reward functions by generating reward implementations whose utility is assessed through training outcomes ( 20 ; 9 ) . Related ideas have also been explored for program and code evolution within genetic programming frameworks ( 11 ; 4 ) , as well as for broader evolutionary computation, such as population-based and quality-diversity search in neural architecture optimization ( 21 ) . However, across these lines of work, evolutionary search typically targets individual components of learning systems, while the reinforcement-learning update rule itself remains fixed. In this work, we investigate evolutionary search with large language models to discover novel reinforcement-learning algorithms by directly searching over learning update rules. Our approach builds on REvolve ( 9 ) , which formulates reward-function discovery as an evolutionary process using island populations ( 3 ) and language-model-based variation. We apply the same evolutionary structure to a different search space. However, instead of evolving reward functions, we evolve learning update rules as executable code under a fixed policy architecture, optimizer, and training configuration. In this setting, each candidate update rule defines a distinct reinforcement-learning algorithm. To encourage the synthesis of noncanonical learning rules and to assess whether genuinely new update rules can emerge, we prohibit the use of standard mechanisms such as actor–critic decomposition ( 16 ) , bootstrapped value targets ( 33 ) , and temporal-difference error computation ( 33 ) . Each candidate algorithm is evaluated by full training on a fixed set of environments, and selection is based on aggregated empirical performance. We further extend REvolve with a post-evolution hyperparameter optimization applied to top-performing algorithms to evaluate robustness and improve stability under fixed training budgets. An overview of the proposed method is illustrated in Figure 1 . In summary, the main contributions of this paper are as follows: • We formulate reinforcement-learning algorithm discovery as an evolutionary search process over executable learning update rules, enabling the direct synthesis of complete learning algorithms. • We extend REvolve to search in the update-rule space using LLM-guided mutation and crossover, with explicit constraints that exclude canonical RL mechanisms to promote the discovery of nonstandard algorithms. • We empirically demonstrate that the proposed framework discovers new reinforcement-learning algorithms that achieve competitive performance across multiple Gymnasium benchmarks after post-evolution hyperparameter optimization. Figure 1 . Illustrative overview of the proposed method. A population of candidate algorithms is iteratively evolved. In each generation (solid arrows), a large language model proposes coherent variants ( \mathcal{L}_{f_{1}},\mathcal{L}_{f_{2}},\ldots,\mathcal{L}_{f_{n}} ), which are evaluated via training in Gymnasium environments to obtain fitness scores ( \bar{F}^{(g)}_{k} ) that are subsequently used for selection and population updates. Post-evolution (dashed arrows), a hyperparameter setting ( \beta_{i}^{\star} ), selected via LLM-guided optimization, optimizes the resulting best update rule ( \mathcal{L}^{\star} ), which is then advanced for final evaluation. 2. Related Work Several lines of work have focused on automating components of reinforcement learning systems that are external to the learning update rule itself. This includes neural architecture search for reinforcement learning agents ( 36 ; 27 ; 31 ) , large-scale hyperparameter optimization frameworks targeting training stability and sample efficiency ( 13 ; 35 ) , and methods that automate the design of auxiliary objectives or training signals ( 14 ; 10

Read the complete text