Vintix: Action Model via In-Context Reinforcement Learning

Andrei Polubarov, Nikita Lyubaykin, Alexander Derevyagin, Ilya Zisman, Denis Tarasov, Alexander Nikulin, Vladislav Kurenkov · 2025

Read Vintix: Action Model via In-Context Reinforcement Learning by Andrei Polubarov, Nikita Lyubaykin, Alexander Derevyagin, Ilya Zisman, Denis Tarasov, Alexander Nikulin, Vladislav Kurenkov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19400.txt.

Introduction

Andrey Polubarov and co-authors introduce Vintix, an action model that learns at test time through trial and error, in a 2025 paper. The idea borrows from how language models adapt to context, but here the goal is to gain reward. Scaling this in-context learning past toy tasks and single domains has been an open problem.

The team build one fixed model that works across domains and learns new behavior from its own interaction history. They use a method that distills learning algorithms into the model itself. Their results suggest this route can rival learning from expert demos when building general agents. Code was released with the work. The paper marks early steps toward agents that improve as they act.

Catalog record

CREATORAndrei Polubarov, Nikita Lyubaykin, Alexander Derevyagin, Ilya Zisman, Denis Tarasov, Alexander Nikulin, Vladislav Kurenkov
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-19400

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vintix: Action Model via In-Context Reinforcement Learning Andrei Polubarov, Nikita Lyubaykin, Alexander Derevyagin, Ilya Zisman, Denis Tarasov, Alexander Nikulin, Vladislav Kurenkov Abstract In-Context Reinforcement Learning (ICRL) represents a promising paradigm for developing generalist agents that learn at inference time through trial-and-error interactions, analogous to how large language models adapt contextually, but with a focus on reward maximization. However, the scalability of ICRL beyond toy tasks and single-domain settings remains an open challenge. In this work, we present the first steps toward scaling ICRL by introducing a fixed, cross-domain model capable of learning behaviors through in-context reinforcement learning. Our results demonstrate that Algorithm Distillation, a framework designed to facilitate ICRL, offers a compelling and competitive alternative to expert distillation to construct versatile action models. These findings highlight the potential of ICRL as a scalable approach for generalist decision-making systems. Code to be released at dunnolab/vintix . † † affiliationnotice: Equal contribution 1 Introduction Figure 1 : Vintix: Self-Correction with Many-Shot ICRL. Many-shot ICRL consistently self-corrects online on the training tasks (87) nearing the demonstrators‘ performance across four domains. Optimal number of shots for many-shot ICRL are shown inside the bar for each task. One shot corresponds to one online episode. See Section 3 for more details and comparisons to action models based on expert distillation. The pursuit of generalist control and decision-making agents has long been a major target of the reinforcement learning (RL) research community ( Sutton & Barto, 1998 ) . These agents are envisioned to handle a diverse set of tasks while exhibiting adaptation properties such as self-correction and self-improvement based on reward functions. Notably, traditional online RL algorithms demonstrate these properties in narrow domains and can achieve exceptional performance ( Berner et al., 2019 ; Badia et al., 2020 ; Schrittwieser et al., 2020 ; Baker et al., 2022 ; Team et al., 2023 ) . However, their online nature of learning and frequent reliance on environment-specific training is still considered a challenge to overcome limiting the scalability ( Dulac-Arnold et al., 2019 ; Levine et al., 2020 ) . In contrast to online reinforcement learning, a recent boom of generative models, revitalized by Large Language Models and their emergent properties ( Brown et al., 2020 ) , sparked a high amount of interest in leveraging offline data for RL training: from specialized offline RL algorithms ( Levine et al., 2020 ; Tarasov et al., 2024 ) to Action Models capable of handling many cross-domain tasks at the same time ( Reed et al., 2022 ; Gallouédec et al., 2024 ; Haldar et al., 2024 ; Collaboration et al., 2024 ; Schmied et al., 2024 ; Sridhar et al., 2024 ) . Figure 2 : Vintix: Approach Overview. Stage 1 (Noise Distillation) - approximating policy improvement trajectory by injecting gradually annealed uniform noise (see Algorithm 1 ). Stage 2 (Cross-Domain Dataset) - combining collected multi-episodic sequences into shared cross-domain dataset for subsequent model training. Stage 3 (Algorithm Distillation) - running AD ( Laskin et al., 2022 ) with collated \{s,a,r\} triplets on collected dataset. Current approaches to pre-training cross-domain action models broadly fall into two categories. The first utilizes all available data, conditioning policies on return-to-go targets in a framework inspired by upside-down reinforcement learning principles ( Schmidhuber, 2019 ; Lee et al., 2022 ; Schmied et al., 2024 ) . The second, and currently dominant, approach prioritizes expert demonstrations, often disregarding reward signals entirely, as exemplified by Reed et al. (2022) and Haldar et al. (2024) . While the field increasingly steers toward language and demonstration-guided paradigms over reward-centric approaches, the “Reward is enough” principle ( Silver et al., 2021 ) continues to offer a foundational framework for agents that learn to maximize arbitrary rewards through trial and error. This perspective has inspired methods like Algorithm Distillation ( Laskin et al., 2022 ) , which operationalizes reward-driven learning by training agents via next-token prediction over historical reinforcement learning (RL) trajectories — a mechanism that directly enables In-Context Reinforcement Learning (ICRL). Though initial implementations of this framework, alongside extensions such as by Zisman et al. (2024a) ; Zisman et al. (2024b) ; Sinii et al. (2024) ; Nikulin et al. (2024) , have been demonstrated only on toy and grid-based tasks within single domains, their data-driven origin positions them as a promising avenue for scaling reward-centric agents to broader, more complex environments. In this work, we start an investigation into cross-domain action models through In-Context Reinforcement Learning (ICRL), grounded in the Algorithm Distillation framework ( Laskin et al., 2022 ) . By focusing on a data-centric approach to ICRL (summarized in Figure 2 ), we present Vintix, a multi-task action model that exhibits initial signs of inference-time self-correction on training tasks ( Figure 1 ) and preliminary evidence of adaptation to parametric task variations ( Section 3.3 ). Our key contributions are as follows: • Continuous Noise Distillation ( Section 2.1 ): We propose a data collection strategy extending the work of Zisman et al. (2024a) to continuous action spaces to ease the collection of training data. • Open Tools and Datasets for ICRL ( Section 2.2 ): We publicly release datasets for 87 tasks across four domains (Meta-World, MuJoCo, Bi-DexHands, Industrial-Benchmark), along with data collection tools and instrumentation to support the development of action models eliciting ICRL behavior. • Cross-Domain Scaling of ICRL ( Section 3 ): We empirically demonstrate that the proposed model, Vintix, can self-correct to attain demonstrator-level performance on training tasks ( Figure 1 ) and adapt to controlled parametric task variations at inference-time. 2 Approach At the core of our approach ( Figure 2 ) is Algorithm Distillation ( Laskin et al., 2022 ) , a two-step Offline Meta-RL algorithm. The first step involves collecting ordered training histories from base reinforcement learning (RL) algorithms, while the second step involves a decoder-only transformer trained solely for the next-action prediction. This approach facilitates in-context learning by effectively distilling the policy improvement operator into a causal sequence model. We further propose two augmentations to this technique: (1) democratizing the data collection process by introducing a continuous extension of the noise-distillation procedure by Zisman et al. (2024a) ; (2) conducting generalist agent-style cross-domain training on the acquired dataset. 2.1 Continuous Noise Distillation Collecting learning histories can be time-consuming and computationally expensive, as it requires curated learning histories of RL algorithms for each task individually. The resulting trajectories may be excessively long due to poor sample efficiency and may exhibit noise due to training instabilities. Recently, Zisman et al. (2024a) demonstrated that learning trajectories approximated through noise distillation can facilitate the emergence of in-context learning capabilities via next-action prediction. In this framework, called AD^{\epsilon} , policy improvement is approximated by gradually reducing the proportion of random noise injected into the demonstrator policy during its execution in the environment. More specifically, at each time step, a random action is selected with probability \epsilon , while a demonstrator action is chosen with probability 1-\epsilon . \epsilon is annealed throughout the trajectory, starting at \epsilon=1 (random policy) at the beginning and gradually decreasing to \epsilon=0 (demonstrator policy) by the end. The AD^{\epsilon} data collection strategy was originally designed for discrete action spaces. In our work, we propose its extension to

Read the complete text