Discovering Reinforcement Learning Interfaces with Large Language Models

Akshat Singh Jaswal, Ashish Baghel, Paras Chopra · 2026

Read Discovering Reinforcement Learning Interfaces with Large Language Models by Akshat Singh Jaswal, Ashish Baghel, Paras Chopra (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03408.txt.

Introduction

Akshat Singh Jaswal, Ashish Baghel, and Paras Chopra study whether language models can discover complete reinforcement learning interfaces from raw simulator state. Building observation mappings and reward functions by hand is a major cost for each new task, and past automation fixed the observations and designed rewards only.

Their LIMEN framework evolves executable interface programs with language model guidance, refining them with policy training feedback and only a trajectory-level success signal. Across gridworld, locomotion, and manipulation domains, evolving observations and rewards together found working interfaces, while optimizing either part alone failed badly on at least one domain.

Catalog record

CREATORAkshat Singh Jaswal, Ashish Baghel, Paras Chopra
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03408

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Discovering Reinforcement Learning Interfaces with Large Language Models Akshat Singh Jaswal, Ashish Baghel, Paras Chopra Abstract Reinforcement learning systems rely on environment interfaces that specify observations and reward functions, yet constructing these interfaces for new tasks often requires substantial manual effort. While recent work has automated reward design using large language models (LLMs), these approaches assume fixed observations and do not address the broader challenge of synthesizing complete task interfaces. We study RL task interface discovery from raw simulator state, where both observation mappings and reward functions must be generated. We propose LIMEN 1 1 1 Code available at https://github.com/Lossfunk/LIMEN , a LLM guided evolutionary framework that produces candidate interfaces as executable programs and iteratively refines them using policy training feedback. Across novel discrete gridworld tasks and continuous control domains spanning locomotion and manipulation, joint evolution of observations and rewards discovers effective interfaces given only a trajectory-level success metric, while optimizing either component alone fails on at least one domain. These results demonstrate that automatic construction of RL interfaces from raw state can substantially reduce manual engineering and that observation and reward components often benefit from co-design, as single-component optimization fails catastrophically on at least one domain in our evaluation suite. 1 Introduction A central challenge in reinforcement learning is specifying the interface through which agents interact with environments, what they observe and how they are rewarded. While progress has been made in learning algorithms, the interface itself remains designed by human experts. Manually engineering these components is a critical bottleneck, as these design choices largely determine an agent’s learning efficiency, exploration, and final policy performance ( 24 ) . Recent work has explored automating reward design, often using large language models (LLMs) to generate reward functions or reward models from task descriptions or environment feedback ( 29 , 13 , 14 ) . However, these approaches assume a fixed already tuned observation interface. In many environments, the raw state representations contain poorly structured information that can hinder learning, while carefully designed observations can substantially simplify the learning problem. Despite its importance, automatic interface discovery has been relatively underexplored compared to reward design. In this work, we address the problem of RL interface discovery by jointly optimizing the observation mapping and reward function. We formalize this interface as a pair (\phi,R) , where \phi maps environment states to observations and R specifies the reward function; together, these induce the effective Markov Decision Process (MDP) experienced by the agent. We assume access to a trajectory-level success metric that evaluates whether the task was completed for example, whether the agent reached the goal or maintained tracking error below a threshold. This metric serves as the fitness signal for evolutionary search but is distinct from the per-step reward function, which must be discovered. Given this signal, we propose LIMEN ( L earning I nterfaces via M DP-guided E volutio N ), a method that discovers effective interfaces using LLM-guided mutation and evolutionary search. By representing \phi and R as executable programs, LIMEN evolves candidate interfaces through a quality-diversity archive, evaluating each by training RL agents to measure performance. Figure 1 illustrates the overall LIMEN framework. Starting from a task description and raw simulator state, the system generates candidate observation and reward programs, evaluates them through policy learning, and iteratively refines the interface using evolutionary selection. Figure 1 : Overview of the LIMEN framework. The outer loop performs evolutionary search: LIMEN selects a parent interface from the MAP-Elites archive, mutates it via LLM-guided code generation, and evaluates the resulting interface by training an RL agent in the inner loop. The interface (\phi,R) mediates between the raw simulator state and the agent, defining the observations and rewards that constitute the induced MDP. Fitness is measured by trajectory-level task success and fed back to update the archive. To evaluate this approach, we design a suite of novel experiments across gridworld reasoning tasks and robotic control environments. Figure 2 illustrates the environments used in our experiments. These experiments specifically test the ability of LIMEN to generate effective interfaces for novel tasks, demonstrating that jointly evolving observations and rewards is the only approach that avoids catastrophic failure across all five tasks, whereas observation-only and reward-only optimization each fail on at least one domain. Analysis of the discovered interfaces further reveals consistent, interpretable patterns in both observation features and reward shaping strategies that emerge through the evolutionary process. 2 Problem Formulation 2.1 RL Interface and Induced MDP We assume access to a simulator world model defined by a Markov decision process (MDP) ( 21 ) \mathcal{M}=(\mathcal{S},\mathcal{A},T,\rho_{0}), where \mathcal{S} is the simulator state space, \mathcal{A} is the action space, T(s^{\prime}\mid s,a) denotes the transition dynamics, and \rho_{0} is the initial state distribution. We assume that a task-specific success metric F:\Pi\rightarrow\mathbb{R} is available, which evaluates the performance of a trained policy \pi over full episodes and serves as the fitness function. An RL interface is defined as a pair \mathcal{I}=(\phi,R), where: • \phi:\mathcal{S}\rightarrow\mathcal{O} is an observation mapping, • R:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R} is a reward function. The interface transforms the simulator into an induced learning problem. Given \mathcal{I} , we define the induced MDP \mathcal{M}_{\mathcal{I}}=(\mathcal{O},\mathcal{A},T_{\phi},R), where observations are given by o_{t}=\phi(s_{t}) and T_{\phi} denotes the observation-level dynamics induced by T . Figure 2 : Evaluation environments. Top: XLand-MiniGrid tasks of increasing compositional complexity— (a) object pickup among distractors, (b) relational placement, (c) multi-step rule chain across rooms. Bottom: MuJoCo tasks— (d) quadruped push recovery, (e) manipulator trajectory tracking. 2.2 Interface Discovery Objective Let \mathcal{A}_{\text{RL}} denote a reinforcement learning algorithm. Given an interface \mathcal{I}=(\phi,R) , we denote by \pi_{\mathcal{I}}=\mathcal{A}_{\text{RL}}(\mathcal{M}_{\mathcal{I}}) the policy obtained by training on the induced MDP. We study the problem of task interface discovery from raw simulator state. The objective is to identify an interface that maximizes task performance under the evaluation metric F : \mathcal{I}^{*}=\arg\max_{\phi,R}\;\mathbb{E}_{\xi}\left[F\big(\pi_{\mathcal{I}}\big)\right], where the expectation is taken over sources of stochasticity \xi , including policy initialization, environment randomness, and training noise. This defines a bilevel optimization problem: \displaystyle\textbf{Outer level:} \displaystyle\max_{\phi,R}\;F(\pi_{\phi,R}) \displaystyle\textbf{Inner level:} \displaystyle\pi_{\phi,R}=\mathcal{A}_{\text{RL}}(\mathcal{M}_{\phi,R}) In this setting, the search space for \phi and R consists of executable programs that operate directly on the raw simulator state variables \mathcal{S} . 3 Related Work Reward Optimization in Reinforcement Learning. A large body of work studies the construction of reward functions for reinforcement learning. Classical approaches include inverse reinforcement learning and imitation learning from demonstrations ( 30 , 6 , 12 ) , as well as preference-based and RLHF methods that infer reward models from human feedback ( 8 , 11 ) . More recently, large language models have been used to automatically generate reward code from natural language task descriptions (e.g., Eureka, Text2Reward, DrEureka) ( 29 , 13 , 14 ) . These approaches optimize the reward function within a fixed environment interface, assuming that the observation space is already sufficient for learning. As a result, these methods cannot address settings where learning fails due to missing or poorly structured observations. A related line of work in inverse reinforcement learning jointly learns observation models alongside reward functions from demonstrations ( 1 , 9 , 4 ) , but does so within policy-learning pipelines that produce neural embeddings over a fixed input space. In contrast, we consider a strictly more general problem: searching

Read the complete text