π RL \pi_{\texttt{RL}} : Online RL Fine-tuning for Flow-based Vision-Language-Action Models

Kang Chen, Zhihao Liu, Tonghe Zhang, Zhen Guo, Si Xu, Hao Lin, Hongzhi Zang, Quanlu Zhang, Zhaofei Yu, Guoliang Fan, Tiejun Huang, Yu Wang, Chao Yu, Equal Contributions † {\dagger} Corresponding Author: Chao Yu [email protected] · 2025

Read π RL \pi_{\texttt{RL}} : Online RL Fine-tuning for Flow-based Vision-Language-Action Models by Kang Chen, Zhihao Liu, Tonghe Zhang, Zhen Guo, Si Xu, Hao Lin, Hongzhi Zang, Quanlu Zhang, Zhaofei Yu, Guoliang Fan, Tiejun Huang, Yu Wang, Chao Yu, Equal Contributions † {\dagger} Corresponding Author: Chao Yu [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-25889.txt.

Introduction

Kang Chen leads thirteen co-authors in a method to fine tune robot models with online reinforcement learning. The models, called vision language action models, let a robot turn images and words into actions. Flow based versions are hard to train with reinforcement learning because the chance of each action, needed for learning, cannot be computed directly from the flow process.

Chen and the team offer two fixes. One turns the denoising steps into a decision process with a learned noise network that allows exact chances. The other merges denoising with interaction through a two layer process for faster exploration. Across benchmarks, reinforcement learning improved results both in familiar and in new settings.

Catalog record

CREATORKang Chen, Zhihao Liu, Tonghe Zhang, Zhen Guo, Si Xu, Hao Lin, Hongzhi Zang, Quanlu Zhang, Zhaofei Yu, Guoliang Fan, Tiejun Huang, Yu Wang, Chao Yu, Equal Contributions † {\dagger} Corresponding Author: Chao Yu [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-25889

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

π RL \pi_{\texttt{RL}} : Online RL Fine-tuning for Flow-based Vision-Language-Action Models Kang Chen, Zhihao Liu, Tonghe Zhang, Zhen Guo, Si Xu, Hao Lin, Hongzhi Zang, Quanlu Zhang, Zhaofei Yu, Guoliang Fan, Tiejun Huang, Yu Wang, Chao Yu, Equal Contributions † {\dagger} Corresponding Author: Chao Yu [email protected] Abstract Vision-Language-Action (VLA) models enable robots to understand and perform complex tasks from multimodal input. Although recent work explores using reinforcement learning (RL) to automate the laborious data collection process in scaling supervised fine-tuning (SFT), applying large-scale RL to flow-based VLAs ( e . g ., π 0 \pi_{0} , π 0.5 \pi_{0.5} ) remains challenging due to intractable action log-likelihoods from iterative denoising. We address this challenge with π RL \pi_{\text{RL}} , an open-source framework for training flow-based VLAs in parallel simulation. π RL \pi_{\text{RL}} implements two RL algorithms: (1) Flow-Noise models the denoising process as a discrete-time MDP with a learnable noise network for exact log-likelihood computation. (2) Flow-SDE integrates denoising with agent-environment interaction, formulating a two-layer MDP that employs ODE-to-SDE conversion for efficient RL exploration. We evaluate π RL \pi_{\text{RL}} on LIBERO and ManiSkill benchmarks. On LIBERO, π RL \pi_{\text{RL}} boosts few-shot SFT models π 0 \pi_{0} and π 0.5 \pi_{0.5} from 57.6% to 97.6% and from 77.1% to 98.3%, respectively. In ManiSkill, we train π RL \pi_{\text{RL}} in 320 parallel environments, improving π 0 \pi_{0} from 41.6% to 85.7% and π 0.5 \pi_{0.5} from 40.0% to 84.8% across 4352 pick-and-place tasks, demonstrating scalable multitask RL under heterogeneous simulation. Overall, π RL \pi_{\text{RL}} achieves significant performance gains and stronger generalization over SFT-models, validating the effectiveness of online RL for flow-based VLAs. https://github.com/RLinf/RLinf https://huggingface.co/RLinf Figure 1: Overview of \pi_{\text{RL}} . \pi_{\text{RL}} , an RL framework featuring Flow-Noise and Flow-SDE two approaches, is designed to enhance the performance and generalization of SFT-aligned flow-based VLAs, represented by the \pi_{0} and \pi_{0.5} . Results in LIBERO demonstrate that \pi_{\text{RL}} achieves significant gains over few-shot SFT, specifically boosting the \pi_{0.5} one-shot SFT performance on the LIBERO-Long benchmark from 43.9% to 94.0%. Results in ManiSkill further show that \pi_{\text{RL}} can support large-scale multitask RL via heterogeneous parallel simulation, which includes a total of 4,352 task combinations. 1 Introduction Vision-Language-Action (VLA) models ( Din et al. 2025 ) have emerged as a leading solution for general-purpose robots, effectively bridging the gap between high-level multimodal reasoning and low-level physical control ( Firoozi et al. 2025 ) . Conditioned on sensor inputs and language commands, VLAs ( Team et al. 2024 ; Kim et al. 2024 ; Black et al. 2024 ; Intelligence et al. 2025 ) can translate abstract instructions into executable robotic actions, thereby enabling intuitive and flexible human-robot interaction. The training methodology for VLAs follows the standard pre-training and supervised fine-tuning (SFT) paradigm as shown in fig. 1 . Building on the pretrained Vision-Language Model (VLM) ( Touvron et al. 2023 ; Beyer et al. 2024 ) , VLAs are fine-tuned on large-scale, heterogeneous human demonstration datasets ( O’Neill et al. 2024 ; Khazatsky et al. 2024 ) , followed by SFT on the target task to align their capabilities with the specific embodiment and environment. However, reliance on SFT introduces a critical challenge: curating large-scale, high-quality expert trajectories is both laborious and costly ( Din et al. 2025 ) , and the models obtained via SFT tend to overfit to expert demonstrations ( Fei et al. 2025 ) . Recent efforts ( Zang et al. 2025 ; Li et al. 2025a ; Tan et al. 2025 ; Liu et al. 2025b ) have explored expanding the VLA training process with reinforcement learning (RL), establishing a pre-training, SFT, and RL paradigm as shown in fig. 1 , allowing VLAs to improve their performance beyond initial expert demonstrations through active environmental interaction and the development of more generalizable policies. However, these RL advances have been largely confined to autoregressive VLAs, featuring OpenVLA ( Kim et al. 2024 ) and OpenVLA-OFT ( Kim et al. 2025 ) , which employ discrete action decoders that generate output in an autoregressive or parallel fashion. This stands in stark contrast to diffusion- or flow-based VLAs, exemplified by the \pi series models \pi_{0} ( Black et al. 2024 ) and \pi_{0.5} ( Intelligence et al. 2025 ) , which generate actions through iterative refinement in flow matching ( Lipman et al. 2022 ) , offering the advantages of generating action chunks in high-frequency and performing highly dexterous tasks ( Black et al. 2024 ) . Consequently, previous VLA-RL algorithms are incompatible with flow-based VLAs, and the fundamental challenge lies in how to characterize a logarithmic likelihood ( Hutchinson 1989 ; Chen et al. 2018 ) for the executed actions. Contribution: We introduce \pi_{\text{RL}} , the first open-source framework for fine-tuning flow-based VLAs \pi_{0} and \pi_{0.5} with parallel online RL. To address the intractable log-likelihood estimation problem in flow matching, we propose two solutions: Flow-Noise and Flow-SDE. Flow-Noise integrates a learnable noise network into the denoising process and models this stage as a discrete-time Markov decision process (MDP) for exact log-likelihood estimation. Flow-SDE converts the ordinary differential equation (ODE) denoising process into a stochastic differential equation (SDE) while maintaining equivalent marginal distributions for exploration, and builds a two-layer MDP that couples the denoising process with policy-environment interaction, along with a hybrid ODE-SDE sampling technique for training acceleration. Given the formulated MDP and the exact log-likelihood computation, \pi_{\text{RL}} undergoes further optimization via the proximal policy optimization (PPO) ( Schulman et al. 2017 ) algorithm. We conduct extensive experiments on the challenging multi-task benchmarks LIBERO ( Liu et al. 2023 ) and high-fidelity simulator ManiSkill ( Tao et al. 2024 ) to evaluate the effectiveness of \pi_{\text{RL}} optimization on the \pi_{0} and \pi_{0.5} models, with comprehensive findings summarized in fig. 1 . Results on LIBERO. \pi_{\text{RL}} demonstrates substantial performance gains over the SFT baselines, with the average success rate of \pi_{0} improving from 57.6% to 97.6%, and \pi_{0.5} from 77.1% to 98.3%. Notably, on the LIBERO-Long task suite, \pi_{\text{RL}} boosts the performance of the \pi_{0.5} one-trajectory SFT model from 43.9% to 94.0%, surpassing the 92.4% performance of the all-trajectories SFT model. Moreover, we compare against group relative policy optimization (GRPO) ( Shao et al. 2024 ) as an alternative policy gradient algorithm, with the comparison showing that PPO consistently outperforms GRPO across all task suites. Results in ManiSkill. We train the policy to pick 16 types of objects and place them on 17 different receptacles in 16 photorealistable scenes, with a total of 4,352 combinations. \pi_{\text{RL}} boosts the average success rate from 41.6% to 85.7% for \pi_{0} and 41.9% to 84.8% for \pi_{0.5} , demonstrating \pi_{\texttt{RL}} ’s ability to support large-scale multi-task RL. Additionally, we also conduct experiments on the SIMPLER benchmark ( Li et al. 2024 ) , the success rate was elevated from 67.2% to 86.7% for \pi_{0} and from 59.24% to 79.1% for \pi_{0.5} . To sum up, our contributions are: • RL for flow-based VLAs. We introduce \pi_{\text{RL}} , the first online RL fine-tuning framework for flow-based \pi -series VLAs, featuring Flow-Noise and Flow-SDE, two distinct technical solutions that allow exact log-likelihood estimation in flow matching. • Superior Performance. We demonstrate significant performance improvements and enhanced generalization of \pi_{\text{RL}} on the multi-task

Read the complete text