InnateCoder: Learning Programmatic Options with Foundation Models

Rubens O. Moraes, Quazi Asif Sadmine, Hendrik Baier, Levi H. S. Lelis · 2025

Read InnateCoder: Learning Programmatic Options with Foundation Models by Rubens O. Moraes, Quazi Asif Sadmine, Hendrik Baier, Levi H. S. Lelis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12508.txt.

Introduction

Rubens Moraes and co-authors present, in a 2025 paper, a system that gives learning agents a head start from human knowledge stored in foundation models. Agents normally begin from a blank slate and slowly earn even obvious skills. That waste slows every new task.

Their system, InnateCoder, extracts ready-made programs for useful sub-skills from foundation models, with no practice runs needed. Those programs act as temporally extended options for the agent. A search then composes them into larger policies for real problems. The contrast with methods that learn options from scratch is the point. Borrowing humanity's stored know-how could compress the slowest phase of agent training.

Catalog record

CREATORRubens O. Moraes, Quazi Asif Sadmine, Hendrik Baier, Levi H. S. Lelis
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-12508

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

InnateCoder: Learning Programmatic Options with Foundation Models Rubens O. Moraes, Quazi Asif Sadmine, Hendrik Baier, Levi H. S. Lelis Abstract Outside of transfer learning settings, reinforcement learning agents start their learning process from a clean slate. As a result, such agents have to go through a slow process to learn even the most obvious skills required to solve a problem. In this paper, we present InnateCoder , a system that leverages human knowledge encoded in foundation models to provide programmatic policies that encode “innate skills” in the form of temporally extended actions, or options. In contrast to existing approaches to learning options, InnateCoder learns them from the general human knowledge encoded in foundation models in a zero-shot setting, and not from the knowledge the agent gains by interacting with the environment. Then, InnateCoder searches for a programmatic policy by combining the programs encoding these options into larger and more complex programs. We hypothesized that InnateCoder ’s way of learning and using options could improve the sampling efficiency of current methods for learning programmatic policies. Empirical results in MicroRTS and Karel the Robot support our hypothesis, since they show that InnateCoder is more sample efficient than versions of the system that do not use options or learn them from experience. 1 Introduction Deep reinforcement learning (DRL) agents typically begin their training process with randomly initialized neural networks. As a result, they must learn from scratch even the most basic skills required to solve a problem. Due to the high sample complexity of DRL algorithms, such tabula rasa learning can be inefficient and time-consuming, which has inspired several research directions aiming at reusing prior computation Zhu et al. 2023b ; Khetarpal et al. 2022 . In this paper, we harness the general human knowledge encoded in foundation models to endow agents with helpful skills before they even start interacting with the environment. This is achieved by using programmatic representations of policies Trivedi et al. 2021 —programs written in a domain-specific language encoding policies—and the foundation models’ ability to write computer programs. Depending on the language used, programmatic policies can generalize better to unseen scenarios and be human-interpretable Verma et al. 2018 ; Bastani et al. 2018 . In addition to these advantages and loosely inspired by the innate abilities of animals Tinbergen 1951 , we show that these representations of policies allow us to harness helpful “innate skills” from foundation models. Given a natural-language description of the problem that the agent needs to learn to solve, our system, which we call InnateCoder , queries a foundation model for programs that encode policies to solve the problem. Although the programs the model generates are unlikely to encode policies that solve the problem, we hypothesize that the set of sub-programs we obtain from these programs can encode helpful temporally extended actions, or options Sutton et al. 1999 . We consider options as functions the agent can call and that will tell it how to act for a number of steps Precup et al. 1998 . Options can improve the agent’s learning in different ways. They can allow the agent to better explore the problem space Machado et al. 2017 or transfer knowledge between different tasks Konidaris and Barto 2007 . This paper presents a novel way of learning options with foundation models. We also present a novel way of using the learned options, which is inspired by recent work on learning semantic spaces of programming languages Moraes and Lelis 2024 . In the semantic space, neighbor programs encode similar but different agent behavior, which can be conducive to algorithms searching for programmatic policies Trivedi et al. 2021 . Instead of searching only in the space of programs induced by the syntax of the language, InnateCoder also searches in the semantic space induced by options. While previous methods can benefit from up to hundreds of options Eysenbach et al. 2019 , InnateCoder ’s use of programmatic options allows it to benefit from thousands of them. InnateCoder ’s approach to harnessing options from foundation models contrasts with previous approaches to automatically learning them, e.g., Tessler et al. 2017 ; Bacon et al. 2017 ; Igl et al. 2020 ; Klissarov and Machado 2023 . This is because options are harnessed from the general knowledge encoded in a foundation model, as opposed to the knowledge the agent gains by interacting with the environment. This zero-shot approach to learning options is enabled by the use of a domain-specific language to bridge the gap between the high-level knowledge encoded in foundation models and the low-level knowledge required at the sensorimotor control level of the agent Klissarov et al. 2024 . For example, foundation models trained on Internet data likely encode the knowledge that, to win a match of a real-time strategy game, the player must collect resources and build structures, which will allow for the training of the units needed to win the game. However, the model cannot issue low-level actions in real time to control dozens of units to accomplish this plan. InnateCoder helps bridge this gap by distilling the knowledge of the model into options that can be executed in real time. We evaluated our hypothesis that foundation models can generate helpful programmatic options in the domains of MicroRTS, a challenging real-time strategy game Ontañón 2017 , and Karel the Robot Pattis 1994 , a benchmark for program synthesis and reinforcement learning algorithms Chen et al. 2018 ; Trivedi et al. 2021 . The results in both domains support our hypothesis, since InnateCoder was more sample-efficient than versions of the system that do not use options or learn options from experience. We also show that the policies InnateCoder learns are competitive and often outperform the current state-of-the-art algorithms. InnateCoder is inexpensive because it uses the foundation model a small number of times as a pre-processing step, making it an accessible system to smaller labs and companies. 1 1 1 InnateCoder is available at https://github.com/rubensolv/InnateCoder . 2 Problem Definition \displaystyle\rho:=~ \displaystyle\texttt{ if }h\texttt{ then }a \displaystyle h:=~ \displaystyle\texttt{frontIsClear }|\texttt{ markersPresent } \displaystyle a:=~ \displaystyle\texttt{move }|\texttt{ putMarker }|\texttt{ pickMarker } \rho if h MP then a PM Figure 1: Top: Context-free grammar specifying a simplified version of the domain-specific language for Karel the Robot. Bottom: Abstract syntax tree for if markersPresent then pickMarker , where MP and PM stand for markersPresent and pickMarker , respectively. Karel is a robot acting on a grid, where it needs to accomplish tasks such as collecting and placing markers on different locations of the grid. In this program, Karel will pick up a marker if one is present in its current location on the grid. We consider sequential decision-making problems formulated as Markov decision processes (MDPs) (S,A,p,r,\mu,\gamma) . Here, S is the set of states and A the set of actions. The function p(s_{t+1}|s_{t},a_{t}) is the transition model, which gives the probability of reaching state s_{t+1} given that the agent is in s_{t} and takes action a_{t} at time step t . The agent observes a reward value of R_{t+1} when transitioning from s_{t} to s_{t+1} . The reward value the agent observes is returned by the function r . \mu is the distribution of the initial states of the MDP; states sampled

Read the complete text