Interactive Post-Training for Vision-Language-Action Models

Shuhan Tan, Kairan Dou, Yue Zhao, Philipp Krähenbühl, UT Austin, Nankai University Code & Model: https://ariostgx.github.io/ript_vla/ · 2025

Read Interactive Post-Training for Vision-Language-Action Models by Shuhan Tan, Kairan Dou, Yue Zhao, Philipp Krähenbühl, UT Austin, Nankai University Code & Model: https://ariostgx.github.io/ript_vla/ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17016.txt.

Introduction

Shuhan Tan, Kairan Dou, Yue Zhao, and Philipp Krähenbühl wrote this paper in two thousand twenty-five. It post-trains robot vision, language, and action models by live trial, with only a pass or fail mark as the reward.

Past pipelines lean on piles of expert demos and copy work, which starves new tasks. The frame, RIPT-VLA, rolls out tries in a dynamic way and scores each by how it beats its batchmates. With one demo, a model that could not do a task at all can learn it in a few rounds. Gains hold across tasks and scenes, and the skill stands up to shifts in the start pose. The method fits many base models.

Catalog record

CREATORShuhan Tan, Kairan Dou, Yue Zhao, Philipp Krähenbühl, UT Austin, Nankai University Code & Model: https://ariostgx.github.io/ript_vla/
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-17016

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interactive Post-Training for Vision-Language-Action Models Shuhan Tan, Kairan Dou, Yue Zhao, Philipp Krähenbühl, UT Austin, Nankai University Code & Model: https://ariostgx.github.io/ript_vla/ Abstract We introduce RIPT-VLA , a simple and scalable reinforcement-learning-based interactive post-training paradigm that fine-tunes pretrained Vision-Language-Action (VLA) models using only sparse binary success rewards. Existing VLA training pipelines rely heavily on offline expert demonstration data and supervised imitation, limiting their ability to adapt to new tasks and environments under low-data regimes. RIPT-VLA addresses this by enabling interactive post-training with a stable policy optimization algorithm based on dynamic rollout sampling and leave-on-out advantage estimation. RIPT-VLA has the following characteristics. First, RIPT-VLA applies to various VLA models, resulting in an improvement on the lightweight QueST model by 21.2%, and the 7B OpenVLA-OFT model to an unprecedented 97.5% success rate. Second, RIPT-VLA is computationally efficient and data-efficient: With only one demonstration, RIPT-VLA enables an unworkable SFT model ( 4% ) to succeed with a 97% success rate within 15 iterations. Furthermore, we demonstrate that the policy learned by RIPT-VLA generalizes across different tasks and scenarios and is robust to the initial state context. These results highlight RIPT-VLA as a practical and effective paradigm for post-training VLA models through minimal supervision. Figure 1: Overview of RIPT-VLA . While VLA models are typically trained with two supervised stages, we propose a third stage: Reinforcement Interactive Post-Training for VLA. RIPT-VLA sets state-of-the-art results across diverse benchmarks. It also presents remarkable improvement under the low-data regime, transforming an SFT model from near failure to 97% with one demonstration. 1 Introduction Vision-Language-Action (VLA) models [ 40 ] aim to enable agents to perceive, reason, and act in the physical world with a unified interface. Current VLA models are trained with two supervised stages: large-scale pretraining on diverse human demonstrations, followed by supervised fine-tuning (SFT) on smaller-scale task-specific datasets. This paradigm has some distinct advantages: Pre-training enables the VLA model to build general visuomotor skills while SFT allows it to specialize in specific environments [ 15 ] . Supervised training allows VLA models to learn from large-scale pre-recorded vision-language-action datasets. However, this supervised approach also has two core limitations: First, data are collected offline. The VLA model learns to imitate interactions with the environment, but never sees the consequences of its own actions. As a result, the learned policy often fails to handle the complexities of real-world scenarios, especially for long-horizon tasks. Second, task-specific SFT via imitation learning relies heavily on large-scale, high-quality human demonstrations. These data are expensive and time-consuming to collect, and performance degrades significantly when only a small number of demonstrations are available. In this paper, we propose RIPT-VLA : a third stage for VLA training paradigm with R einforcement I nteractive P ost- T raining. After pretraining and supervised fine-tuning, we allow the VLA model to interact with the multitask environment and receive binary success/failure rewards. We then optimize the VLA model to directly improve its success rate across multiple tasks through reinforcement learning. Inspired by prior RL frameworks for LLMs reasoning [ 9 ] , we propose a stable and efficient RL framework for VLA finetuning in Section 4 . Specifically, we extend the LOOP framework [ 4 ] which combines REINFORCE leave-one-out (RLOO) advantage estimation [ 16 ] and proximal policy optimization (PPO) [ 28 ] . Unlike LOOP, we construct uniform batches of non-zero advantage samples, filtering out any group of trajectories with zero-advantage, and sampling rollouts until sufficient samples exist. This uniform batch construction leads to improved training stability, especially as training progresses and the VLA becomes more successful. RIPT-VLA allows efficient and stable VLA policy update without relying on shaped or learned rewards, or critic models. Using Reinforcement Learning in a third training stage has a few distinct advantages: It is more data efficient, yielding close to state-of-the-art performance with only a single SFT demonstration. The resulting VLA model achieves a much higher performance on the end-task, as it gets to see interactions with the environment during training. RIPT-VLA works with both tokenized [ 22 ] and continuous actions [ 14 ] . RIPT-VLA resonates with the recent trend of paradigm shift in LLM training [ 24 , 9 ] . While pretraining on large-scale text corpora equips LLMs with broad knowledge and powerful skills, they often struggle with challenging tasks that require precise reasoning, multi-step planning, or tool using [ 34 ] . To address these limitations, reinforcement learning has emerged as a critical post-training stage to reactivate and steer pretrained knowledge with only a small amount of interactive feedback [ 24 ] . Similarly, we observe that pretrained VLA models also encode rich visuomotor skills, yet struggle to apply them effectively for new tasks and scenarios. RIPT-VLA bridges this gap by using only sparse binary rewards to unlock these latent skills with a small number of optimization steps. Through comprehensive experiments in Section 5 , we demonstrate that RIPT-VLA achieves state-of-the-art results when combined with both large-scale and lightweight VLA models across a diverse set of tasks. On the LIBERO benchmark [ 21 ] , RIPT-VLA improves QueST [ 22 ] , the best lightweight VLA model, on all four task suites by 10.9% absolute success rate (SR) on average (Table 1 ). When evaluated on OpenVLA-OFT [ 14 ] , the best-performing large VLA model with an already high success rate (96.7%), RIPT-VLA still helps by further reducing the failure rate from 3.3% to 2.5%. We also achieve top performance on many-task benchmarks LIBERO-90 (94.3%) and MetaWorld45 [ 36 ] ( 92.2% ), showing the effectiveness of RIPT-VLA in improving multi-task (up to 90) performance with a single model (Table 2 ). Most notably, in the extreme low-data regime with only a single training demo, RIPT-VLA adapts pretrained knowledge to new task goals or scenarios with remarkable efficiency, boosting success rate from below 4% to over 97% within only 15 RL iterations. 2 Related Works Vision-Language-Action Models. Vision-Language-Action (VLA) models empower embodied agents to interpret multimodal inputs—such as visual observations and natural-language instructions—and translate them into meaningful actions within the physical world [ 40 ] . Seminal works like RT-2 [ 40 ] , RT-1 [ 2 ] , PaLM-E [ 7 ] , Octo [ 32 ] , Dita [ 11 ] , \pi_{0} [ 1 ] , and \pi_{0.5} [ 13 ] , together with OpenVLA [ 15 ] , showcase VLAs achieving emergent semantic reasoning and generalization to novel tasks and environments. These models are typically developed through a two-stage supervised-learning paradigm that begins with an initial pre-training phase on extensive, web-scale datasets [ 7 , 3 ] , which is crucial for acquiring generalizable visuomotor skills, grounding language in perception, and building robust internal representations. While this two-stage approach has advanced the field, its offline nature imposes key limitations. The supervised fine-tuning (SFT) stage typically requires vast expert demonstrations for new tasks or environments, thereby degrading few-shot performance. This highlights a critical gap: the need for methods that adapt pretrained VLAs beyond static imitation by leveraging interactive experience and reducing reliance on extensive expert data. Reinforcement Learning for Large Language Models. Large Language Models (LLMs) offer a precedent for enhancing pretrained models. While LLMs gain broad capabilities via pre-training and SFT,

Read the complete text