FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation
Shuyi Zhang, Yunfan Lou, Hongyang Cheng, Yichen Guo, Chuyao Fu, Yaoxu Lyu, Xiaojie Zhang, Haoran Li, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang 🖂 · 2026
Read FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation by Shuyi Zhang, Yunfan Lou, Hongyang Cheng, Yichen Guo, Chuyao Fu, Yaoxu Lyu, Xiaojie Zhang, Haoran Li, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang 🖂 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26006.txt.
Introduction
Shuyi Zhang leads a large team improving how vision-language-action robot models learn from practice. Such models copy demonstrations, so flawed training data caps their skill. Reinforcement learning can push past that ceiling, but it learns slowly and unstably. Early training often unlearns good behaviour because the value function starts out unreliable. Exploration data can also be too poor to learn from.
FORCE answers with three stages. A value-calibrated warm-up uses on-policy rollouts to fix the value function's shift. During online learning, that calibrated function filters action proposals. It also filters expert data, so only high-value actions update the policy. Across simulated and real tasks, success rates jump by seventy-nine points. Prior reinforcement methods trail by ten points, and training runs about a third faster. No human intervention is needed, and the usual early success drop disappears.
Catalog record
| CREATOR | Shuyi Zhang, Yunfan Lou, Hongyang Cheng, Yichen Guo, Chuyao Fu, Yaoxu Lyu, Xiaojie Zhang, Haoran Li, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang 🖂 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.26006
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation Shuyi Zhang, Yunfan Lou, Hongyang Cheng, Yichen Guo, Chuyao Fu, Yaoxu Lyu, Xiaojie Zhang, Haoran Li, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang 🖂 Abstract Vision-Language-Action (VLA) models are often constrained by the ’imitation ceiling’ imposed by sub-optimal data. While Reinforcement Learning (RL) fine-tuning can surpass this limit, it is notoriously sample inefficient. This challenge arises from two core issues: (1) catastrophic initial unlearning due to an unstable Q-function and (2) inefficient policy updates caused by low-quality exploration data, often forcing a reliance on costly human interventions. We introduce FORCE , a 3-stage framework that stabilizes fine-tuning by tackling both issues. FORCE first incorporates a Value-Calibrated Warm-Up phase, utilizing on-policy rollouts to mitigate the distributional shift of the Q-function. Subsequently, during the online stage, this calibrated Q-function acts as a filter for both the policy’s own action proposals and expert data, ensuring only high-value actions are used for the policy update. We evaluate FORCE on various simulation and real-world tasks, and the result shows that FORCE achieves a 79% absolute improvement in success rates and outperform prior RL methods by 10%, while accelerating training by 32.5%. Critically, it mitigates the common success rate drop and achieves this robust performance without human intervention, marking a significant step towards deploying capable and autonomous robotic agents. † † affiliationnotice: Equal contribution 🖂 Corresponding author 1 Introduction Figure 1 : Overview of the FORCE framework. Our method employs a three-stage reinforcement fine-tuning pipeline that progressively calibrates value estimation and stabilizes policy improvement: (1) offline Cal-QL pretraining to obtain a conservative and well-grounded critic, (2) mixed-rollout value pre-calibration to bridge offline and online distributions and mitigate O2O drift, and (3) online fine-tuning with balanced expert/policy replay coupled with VGPD, enabling safe and sample-efficient policy refinement beyond the imitation ceiling. Vision-Language-Action models have demonstrated remarkable potential in executing general-purpose robotic tasks ( 15 ; 39 ; 8 ) . These models are typically pre-trained through Imitation Learning (IL), enabling robots to understand complex instructions and map them to physical actions ( 2 ; 1 ) . However, pure imitation learning (such as Supervised Fine-Tuning, SFT) faces a fundamental bottleneck: the policy’s performance is constrained by the quality of the demonstration data, known as the ”imitation ceiling” ( 30 ) . This is not merely an empirical observation but a fundamental statistical limitation, as imitation learning suffers from error compounding that scales quadratically with task horizon ( 29 ) . Since human demonstration data inevitably contains sub-optimal and even inconsistent actions ( 11 ; 4 ) , which standard IL struggles to learn from without additional information like trajectory rankings ( 9 ) , the performance ceiling of VLA models is firmly capped. Reinforcement learning offers a clear path for VLA models to surpass the limitations of demonstration data through online interaction with the environment, thereby breaking the ”imitation ceiling” ( 36 ; 13 ) . Indeed, recent works have shown that even a small amount of online fine-tuning can dramatically improve the performance of large-scale, pre-trained IL policies ( 19 ; 12 ; 18 ) . Despite this promise, applying RL directly to the online fine-tuning of VLA models faces severe challenges. The most prominent issue is that RL fine-tuning is notoriously sample inefficient in real-world physical interactions ( 10 ) . This sample inefficiency remains the primary bottleneck, even as recent reinforced fine-tuning (RFT) frameworks have shown impressive gains with just minutes of online interaction ( 5 ; 35 ) . The root of this inefficiency can be traced to two core difficulties in offline-to-online RL on real robots. (1) Catastrophic ’Initial Unlearning’ : When transitioning from offline pre-training (which often uses conservative value estimates) to online fine-tuning, policies suffer from a well-documented performance collapse ( 26 ; 25 ) . This ”initial unlearning” is caused by a Q-value scale mismatch, where the highly underestimated offline Q-function is ”deceived” by new online data, leading to a catastrophic adjustment period and negating the benefits of pre-training. (2) Inefficient Policy Updates : The policy update process is severely hindered by low-quality exploratory data (i.e., the policy generates a large number of useless actions when exploring unfamiliar regions). To address these problems and ensure safe exploration, existing methods ( 23 ; 5 ; 14 ) are often forced to rely on costly and difficult-to-scale Human-in-the-Loop (HiL) interventions ( 23 ) , which treat human attention as a scarce resource. To address these challenges, we introduce FORCE, an intervention-free framework that bridges the gap between static datasets and dynamic physical interaction. To mitigate catastrophic unlearning, FORCE incorporates a distributional warm-up phase that calibrates the value function on the policy’s visitation distribution prior to any actor updates. This aligns the critic’s support with the agent’s current behavior, ensuring stable value estimates during the offline-to-online transition. Subsequently, the framework employs a value-guided self-distillation mechanism to govern online fine-tuning. Acting as a dynamic filter, this module projects the policy onto a target distribution derived strictly from high-value transitions, effectively eliminating exploration noise and ensuring monotonic improvement without human supervision. We evaluated FORCE on several challenging robotic manipulation tasks both in simulation and the real-world. The experimental results demonstrate that FORCE achieves significant performance gains, reaching nearly 100% success rates on several tasks, all without human intervention. Our main contributions are summarized as follows: 1. FORCE Framework: A principled three-stage pipeline for intervention-free offline-to-online RL fine-tuning starting from pre-trained models (e.g., VLA). 2. Distributional Warm-up: A mechanism to mitigate cold-start covariate shift by aligning Q-function support with policy visitation prior to fine-tuning. 3. Value-Guided Policy Distillation: A theoretically grounded regularized policy improvement operator that filters high-variance exploration to accelerate convergence. 4. SOTA Performance and Efficient Adaptation: Empirical validation on physical robots showing that FORCE achieves near-perfect success rates on contact-rich tasks with significantly improved sample efficiency compared to baselines. 2 Related Work Vision-Language-Action Models Vision-Language-Action (VLA) Models have recently enabled robots to perceive, reason, and act using multimodal inputs ( 3 ; 39 ) . Early architectures treated manipulation as a discrete token generation problem, like OpenVLA ( 15 ) , often resulting in coarse control resolution. This limitation motivated models to utilize flow matching or diffusion processes to chunk continuous motions into high-dimensional latent flows, such as Octo ( 8 ) , Rdt-1b ( 21 ) and \pi -series models ( 2 ; 1 ; 28 ) . However, these models predominantly rely on supervised imitation learning from static datasets, which prevents them from correcting execution errors or adapting to dynamic environments. In our work, we focus on fine-tuning high-capacity VLA policies using RL to further enhance their performance and generalization. RL Finetuning for VLA Models and Real-world RL Many works recently have explored reinforcement learning algorithm to improve the generalization ability of VLA models. Approaches building on PPO ( 31 ) or GRPO ( 32 ) s with transformer backbones remain largely restricted to simulation ( 19 ; 22 ; 20 ) , as their on-policy nature suffers from poor sample efficiency that prohibits physical deployment. Moreover, these standard formulations are structurally limited to autoregressive backbones, failing to accommodate continuous policies required for dexterous manipulation. Real-world RL training strategies for VLA models