Seeing Realism from Simulation: Efficient Video Transfer for Vision-Language-Action Data Augmentation
Chenyu Hui, Xiaodi Huang, Siyu Xu, Yunke Wang, Shan You, Fei Wang, Tao Huang, Chang Xu · 2026
Read Seeing Realism from Simulation: Efficient Video Transfer for Vision-Language-Action Data Augmentation by Chenyu Hui, Xiaodi Huang, Siyu Xu, Yunke Wang, Shan You, Fei Wang, Tao Huang, Chang Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02757.txt.
Introduction
Chenyu Hui and seven co-authors present a method that turns simulated robot videos into realistic training videos for vision-language-action models. Simulated data are cheap and easy to gather in parallel, but they look unlike the real world and cover few settings, so robots trained on them generalize weakly. Real videos are costly, which is the bottleneck the method attacks.
The pipeline reads structure from simulation through segmentation and captioning, rewrites captions to vary the setting, and renders realistic video with a conditional transfer model, keeping task meaning and action paths intact. A feature-reuse trick speeds generation, and a coreset choice picks a compact, non-redundant set to augment when compute is limited. On several robot benchmarks and a real platform, gains were consistent.
Catalog record
| CREATOR | Chenyu Hui, Xiaodi Huang, Siyu Xu, Yunke Wang, Shan You, Fei Wang, Tao Huang, Chang Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.02757
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Seeing Realism from Simulation: Efficient Video Transfer for Vision-Language-Action Data Augmentation Chenyu Hui, Xiaodi Huang, Siyu Xu, Yunke Wang, Shan You, Fei Wang, Tao Huang, Chang Xu Abstract Vision-language-action (VLA) models typically rely on large-scale real-world videos, whereas simulated data, despite being inexpensive and highly parallelizable to collect, often suffers from a substantial visual domain gap and limited environmental diversity, resulting in weak real-world generalization. We present an efficient video augmentation framework that converts simulated VLA videos into realistic training videos while preserving task semantics and action trajectories. Our pipeline extracts structured conditions from simulation via video semantic segmentation and video captioning, rewrites captions to diversify environments, and uses a conditional video transfer model to synthesize realistic videos. To make augmentation practical at scale, we introduce a diffusion feature-reuse mechanism that reuses video tokens across adjacent timesteps to accelerate generation, and a coreset sampling strategy that identifies a compact, non-redundant subset for augmentation under limited computation. Extensive experiments on Robotwin 2.0, LIBERO, LIBERO-Plus, and a real robotic platform demonstrate consistent improvements. For example, our method improves RDT-1B by 8% on Robotwin 2.0, and boosts π 0 \pi_{0} by 5.1% on the more challenging LIBERO-Plus benchmark. Code is available at: CODE . † † affiliationnotice: ∗ Equal contribution. Work was done during internship at SJTU. 1 Introduction The advent of large-scale Vision-Language-Action (VLA) models has marked a significant milestone in robotics ( 29 ; 4 ; 16 ; 15 ; 29 ; 50 ; 45 ) , enabling robots to interpret natural language instructions and execute complex manipulation tasks. These models typically rely on extensive datasets of real-world robotic trajectories to learn generalizable policies. However, the collection of such real-world data is inherently costly, time-consuming, and difficult to scale, presenting a major bottleneck for broader development and deployment. In contrast, simulated data offers a highly inexpensive alternative. But as noted in recent studies, models that achieve near-perfect performance within a specific simulation benchmark often fail catastrophically when faced with minor perturbations in object layout, lighting, camera viewpoint, or instruction phrasing in scenarios with disturbances ( 13 ; 52 ; 46 ; 32 ) , revealing that many models merely memorize training sequences rather than learning robust, semantic task understanding. Figure 1 : Overall framework of the proposed method. Given a large-scale simulation training set, we introduce a coreset sampling algorithm to select important and diverse samples, which are then augmented to realistic video strips and used for training. We address the critical lack of visual and environmental diversity in robotic training data by proposing an efficient, scalable video augmentation framework. Our approach transforms source video sequences into visually diverse training videos while strictly preserving underlying task semantics and action trajectories. The pipeline first extracts structured conditions from simulation, diversifies the environmental context, and then synthesizes high-fidelity videos using a conditional video diffusion model. To mitigate the substantial computational cost of video transfer, we propose accelerating the diffusion generation by caching velocity during the denoising process. We design adaptive strategies that optimize the trade-off between quality and efficiency. Consequently, this velocity caching mechanism achieves over a 60% reduction in generation time while maintaining high model accuracy. Furthermore, a coreset sampling method is designed to achieve the efficient scalability by only augmenting a subset in the large-scale dataset. For the sake of sampling diverse trajectories that maximize the performance, we build a graph structure that balances the difficulty and diversity among data nodes, where the difficulty and diversity are estimated by a video embedding model and a pretrained VLA policy. By prioritizing samples that are both challenging and highly-diverse, we ensure the significance of the augmentation effect in large-scale datasets. We conduct extensive experiments across multiple benchmarks, including Robotwin 2.0 ( 8 ) , LIBERO ( 26 ) , and the more challenging LIBERO-Plus ( 13 ) to demonstrate the effectiveness of our framework. We also conduct real-world robotic experiments. In summary, our contributions are threefold: • We present an end-to-end video transfer named efficient transfer framework that effectively bridges the sim-to-real gap for VLA data augmentation. • We design a segmented (three-stage) velocity caching strategy, which better matches the dynamics of conditional video generation. It is proved that adopting this strategy can save over 60% transferring time. • We propose a trajectory-level coreset formulation that combines policy difficulty (measured by the policy loss of RDT-1B), and visual diversity (measured via Cosmos-Embed1 representations), leading to more effective selection of high-value training trajectories. 2 Related Work 2.1 Vision-Language-Action Models Since the development of VLM and its been pivotal in advancing robotic control by providing rich multi-modal representations. Recent VLA models are finetuned on large scale robotic trajectory data in an end to end mode to obtain more generalizable policies ( 6 ; 10 ) . Widely used OpenVLA ( 19 ) is based on prismatic ( 18 ) . RDT combines the advantage of diffusion model and transformer ( 29 ) . ACT ( 50 ) introduced chunking for actions based on transformer. The OpenPi series like \pi_{0} ( 4 ) , \pi_{0.5} ( 16 ) and the newest \pi_{0.6} ( 15 ) gains significant performance superiority by enhancing existing VLMs with a dual-system VLA architecture, retaining the text-generation ability of the VLM, and combining with reinforcement learning, respectively. 2.2 Video Generation Recent advances in diffusion-based video generation and style transfer have established this as a pivotal area in multimodal modeling, building upon foundational works like Stable Diffusion ( 36 ) , Stable Video Diffusion ( 5 ) . Contemporary research has expanded these frontiers: Sora ( 7 ) incorporates a temporal VAE and Transformer backbone; Wan ( 40 ) and VACE ( 17 ) employ mixture-of-experts architectures; while NVIDIA’s Cosmos series ( 2 ) , particularly Cosmos Transfer ( 3 ) , provides a robust foundation for world model-based video transfer. Concurrently, generative techniques are increasingly applied to embodied data, with methods like Rebot ( 12 ) , EgoDemoGen ( 47 ) , EMMA ( 9 ) , Embodied Dreamer ( 41 ) , Gigaworld-0 ( 39 ) , RoboTransfer ( 28 ) and Gigabrain-0 ( 38 ) demonstrating effective real-to-sim and sim-to-real data generation. 2.3 Data Centric AI and Efficient Training Data-centric approaches have proven crucial for enhancing model performance and training efficiency, evolving from early heuristic-based filtering methods like CCNet ( 43 ) and C4 ( 34 ) to sophisticated techniques emphasizing deduplication ( 21 ) and semantic redundancy elimination ( 1 ) . Beyond quality filtering, data diversity plays an equally vital role in robustness, as demonstrated by The Pile’s emphasis on heterogeneous data ( 14 ) and Falcon Series’ success with web-scale curation ( 33 ) . Domain-specific selection methods like DSIR ( 44 ) leverage importance resampling for target distribution alignment, while theoretical advances ( 37 ) show intelligent pruning can achieve exponential efficiency gains. This paradigm culminates in \mathbb{D}^{2} Pruning ( 30 ) , which advocates balanced coreset selection based on complementary diversity-difficulty criteria. 3 Method We propose a VLA data augmentation framework to improve sim-to-real generalization. We first analyze the generalization failures of simulation-trained models, then introduce an end-to-end pipeline that generates realistic videos from semantic and structural cues. To scale augmentation efficiently, we develop a