RL + Transformer = A General-Purpose Problem Solver

Micah Rentschler, Jesse Roberts · 2025

Read RL + Transformer = A General-Purpose Problem Solver by Micah Rentschler, Jesse Roberts (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-14176.txt.

Introduction

Micah Rentschler and Jesse Roberts show that a pre-trained Transformer, fine-tuned with reinforcement learning, can learn to solve new problems on its own, in a paper from 2025. The model practices across many episodes of varied tasks. Over time it gains a skill the authors call in-context reinforcement learning.

The trained model solved tasks it had never seen, using few attempts. It also did well on tasks outside its training range. Its gains did not depend much on the quality of its training examples. It could stitch together behaviors shown in its context and adapt when the rules of a task changed mid-stream. The authors present it as a general-purpose problem solver that improves its own answers.

Catalog record

CREATORMicah Rentschler, Jesse Roberts
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-14176

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RL + Transformer = A General-Purpose Problem Solver Micah Rentschler, Jesse Roberts Abstract What if artificial intelligence could not only solve problems for which it was trained but also learn to teach itself to solve new problems (i.e., meta-learn)? In this study, we demonstrate that a pre-trained transformer fine-tuned with reinforcement learning over multiple episodes develops the ability to solve problems that it has never encountered before —an emergent ability called In-Context Reinforcement Learning (ICRL) . This powerful meta-learner not only excels in solving unseen in-distribution environments with remarkable sample efficiency, but also shows strong performance in out-of-distribution environments. In addition, we show that it exhibits robustness to the quality of its training data, seamlessly stitches together behaviors from its context, and adapts to non-stationary environments. These behaviors demonstrate that an RL-trained transformer can iteratively improve upon its own solutions, making it an excellent general-purpose problem solver . 1 Introduction Imagine a Mars mission in which a robot’s appendage malfunctions and becomes unusable. An adaptive agent controlling the robot could relearn how to operate without the appendage, allowing the mission to be completed successfully. This kind of adaptability is observed in nature, where animals adjust to the loss of limbs Cully et al. 2014 . Developing artificial intelligence (AI) systems with similar adaptability and problem-solving abilities in real, non-stationary environments remains a significant challenge. Reinforcement learning (RL) has had great successes solving stationary control problems Sutton and Barto 2018 . However, the application of RL to the real world has been fraught with challenges. Compared to humans, RL methods suffer from low sample efficiency Tsividis et al. 2017 ; Duan et al. 2016 , meaning that they require a vast number of interactions with the environment before learning an effective policy. This inefficiency arises because they begin tabula rasa —without any prior knowledge of the environment—and explore a wide range of possible actions and states to gather enough information to improve performance. In short, RL excels at providing great solutions to specific, invariant problems where ample experience can be gathered to aid the learning process. However, many real-world problems do not require a specialist; they need a generalist with the ability to adapt to an ever-changing environment after only a few experiences. (a) Early inference (b) Mid inference (c) Late inference Figure 1 : ICRL-trained Llama 3.1 learns to solve an unseen Frozen Lake environment. The trajectories in early ( 1(a) ), mid ( 1(b) ), and late ( 1(c) ) interactions show solution refinement. Mistakes in early inference (e.g., falling into holes) disappear with experience in late inference. Recent advances in other areas of AI have shown a remarkable ability to generalize. The introduction of the transformer architecture Vaswani 2017 enables AI models to efficiently focus on relevant information, even when it is surrounded by a large amount of irrelevant context—a task that has been challenging for traditional fully connected neural networks. This capability has led to an emergent phenomenon called in-context learning (ICL) , as demonstrated by Brown et al. Brown et al. 2020 , where the model learns to perform new tasks without adjusting its internal weights. This ability to generalize and learn new tasks without retraining prompts us to ask: Is it possible to train a transformer to function as a reinforcement learning algorithm, improving its predictions based on a few experiences in its context without any additional weight updates? If so, can it generalize beyond its training data, learn with higher sample efficiency, and solve non-stationary environments—all without additional training ? In this paper we show that: (1) Llama 3.1 8B can teach itself (meta-learn) through in-context experience by training on an RL objective, (2) the ability to meta-learn generalizes beyond the problem space in which it was acquired, (3) the model is robust to variations in the quality of its training data, (4) the model can assemble skills in a piecemeal manner to solve problems, and (5) the model learns to self-adapt when the environment changes. This is the first work to show that a large language model (LLM) trained with the simple Deep Q-Network (DQN) algorithm Mnih et al. 2013 can solve non-stationary problems such as adapting to the loss of a limb. Our results also raise thought-provoking questions: Should the Polyak averaging factor be increased in ICRL? Are parameter-efficient methods feasible and effective? And, what elements might be missing to enhance exploration? The successful integration of ICRL with transformer architectures signals a paradigm shift in RL research. By enabling transformers to meta-learn through in-context experiences, we pave the way for agents that can self-improve without additional training. This advancement represents a significant stride toward developing agents that are general-purpose problem solvers able to navigate the complexities of the real world akin to living organisms. 2 Background and Related Work Existing work (discussed below) has shown that RL-trained transformers are meta-learners. First, we review key works that combine transformers with RL, highlighting their contributions and limitations. Then we extend this literature, demonstrating that ICRL offers additional advantages. Specifically, ICRL-trained transformers possess the ability to combine learned skills in novel ways, learn effectively from suboptimal data, and adapt to non-stationary environments. 2.1 Reinforcement Learning as Sequence Modeling A seminal effort in merging transformers with RL is the Decision Transformer (DT) Chen et al. 2021 , which reimagines RL as a sequence modeling problem. Instead of learning a policy or value function in the traditional sense, DT treats the return-conditioned trajectories as language tokens. The model is trained to predict the next action given a sequence of past states, actions, and the desired return (reward-to-go). By providing high target returns during inference, DT can generate action sequences that aim for high cumulative rewards. This approach demonstrates that transformers can model the dependencies in RL tasks without explicit policy optimization. However, DT relies heavily on the quality and diversity of its training data. Since it models the behavior implicit in the dataset, DT may struggle to generalize beyond the trajectories it has seen, lacking the ability to stitch together novel action sequences to achieve unseen goals. This dependence can limit its performance in environments requiring skill combination or in out-of-distribution scenarios. To address these limitations, Algorithm Distillation (AD) Laskin et al. 2022 proposes training transformers to imitate the behavior of RL algorithms themselves, rather than directly modeling action sequences. By learning from data generated by an RL algorithm, the transformer effectively distills the algorithm’s policy into its weights. Remarkably, AD shows that transformers can be more sample-efficient than the algorithms they imitate, suggesting that they capture underlying patterns and structures facilitating efficient learning in a particular environment. Building upon DT and AD, the Decision-Pretrained Transformer (DPT) Lee et al. 2024 aims to overcome dependence on future rewards and specific algorithmic behaviors. DPT trains transformers to imitate an action oracle—a function providing optimal actions for given states—allowing the model to learn optimal policies directly. The authors demonstrate that DPT can perform RL effectively and sample-efficiently, possessing the ability to piece together partial trajectories to achieve goals, akin to dynamic programming. This trajectory stitching addresses a significant limitation of DT. However, DPT’s reliance on an action oracle at train time necessitates prior knowledge of the optimal solution, limiting

Read the complete text