Robotic VLA Benefits from Joint Learning with Motion Image Diffusion

Yu Fang, Kanchana Ranasinghe, Le Xue, Honglu Zhou, Juntao Tan, Ran Xu, Shelby Heinecke, Caiming Xiong, Silvio Savarese, Daniel Szafir, Mingyu Ding, Michael S. Ryoo, Juan Carlos Niebles · 2025

Read Robotic VLA Benefits from Joint Learning with Motion Image Diffusion by Yu Fang, Kanchana Ranasinghe, Le Xue, Honglu Zhou, Juntao Tan, Ran Xu, Shelby Heinecke, Caiming Xiong, Silvio Savarese, Daniel Szafir, Mingyu Ding, Michael S. Ryoo, Juan Carlos Niebles (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18007.txt.

Introduction

Yu Fang leads a twelve author team giving robot control models a sense of motion. Vision language action models turn camera views and spoken orders into robot moves, but they mostly copy expert demos. They rarely reason about how the scene is about to move, which caps their judgment.

The team's design adds a second head that dreams up motion images, maps of where pixels will flow next, while the action head still picks the moves. Training both heads together ties control to motion sense, at no cost to test time speed. Scores rose to 97.5 percent on the LIBERO test set and 58.0 percent on RoboTwin, with a 23 percent gain in real world trials.

Catalog record

CREATORYu Fang, Kanchana Ranasinghe, Le Xue, Honglu Zhou, Juntao Tan, Ran Xu, Shelby Heinecke, Caiming Xiong, Silvio Savarese, Daniel Szafir, Mingyu Ding, Michael S. Ryoo, Juan Carlos Niebles
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-18007

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Robotic VLA Benefits from Joint Learning with Motion Image Diffusion Yu Fang, Kanchana Ranasinghe, Le Xue, Honglu Zhou, Juntao Tan, Ran Xu, Shelby Heinecke, Caiming Xiong, Silvio Savarese, Daniel Szafir, Mingyu Ding, Michael S. Ryoo, Juan Carlos Niebles Abstract Vision-Language-Action (VLA) models have achieved remarkable progress in robotic manipulation by mapping multimodal observations and instructions directly to actions. However, they typically mimic expert trajectories without predictive motion reasoning, which limits their ability to reason about what actions to take. To address this limitation, we propose joint learning with motion image diffusion, a novel strategy that enhances VLA models with motion reasoning capabilities. Our method extends the VLA architecture with a dual-head design: while the action head predicts action chunks as in vanilla VLAs, an additional motion head, implemented as a Diffusion Transformer (DiT), predicts optical-flow-based motion images that capture future dynamics. The two heads are trained jointly, enabling the shared VLM backbone to learn representations that couple robot control with motion knowledge. This joint learning builds temporally coherent and physically grounded representations without modifying the inference pathway of standard VLAs, thereby maintaining test-time latency. Experiments in both simulation and real-world environments demonstrate that joint learning with motion image diffusion improves the success rate of π \pi -series VLAs to 97.5% on the LIBERO benchmark and 58.0% on the RoboTwin benchmark, yielding a 23% improvement in real-world performance and validating its effectiveness in enhancing the motion reasoning capability of large-scale VLAs. 1 Introduction Vision-Language-Action (VLA) models have emerged as a powerful paradigm for generalist robotic manipulation, capable of performing diverse tasks across environments and embodiments 8 ; 77 ; 59 ; 46 ; 32 . These models are typically built upon large-scale Vision-Language Models (VLM) 43 ; 1 ; 62 ; 30 ; 3 , which provide strong perceptual and linguistic understanding and are further equipped with action experts that translate multimodal representations into robot actions. Despite their impressive performance, current VLAs exhibit a fundamental limitation: they lack an explicit mechanism to reason about future dynamics, and largely rely on imitating expert trajectories. This absence of motion-level reasoning restricts their temporal understanding and ultimately limits generalization to new tasks, scenes, and embodiments. Figure 1 : Overview. Our joint learning strategy seamlessly extends existing large-scale VLAs with motion image diffusion – learning motion and action jointly through a shared VLM backbone, enhancing their motion reasoning abilities, and maintaining the same inference pipeline as in standard VLA models. Motion understanding is a fundamental direction of effective visuomotor control. In robotic manipulation, explicit motion representations, such as keypoint traces and optical flow, serve as popular abstractions for describing action-relevant information from manipulation videos 67 ; 74 ; 71 ; 39 ; 54 . Recent studies have incorporated such representations into large-scale VLA models 2 ; 74 ; 34 ; 71 , demonstrating their potential to enhance motion awareness. Meanwhile, world models and video prediction approaches implicitly describe motion through future image prediction based on current observations and have been introduced to guide policy learning 48 ; 7 ; 19 ; 27 ; 4 . Building upon this idea, recent works have explored unified VLA models 36 ; 47 ; 76 ; 65 ; 10 that integrate action models with world models to provide predictive insights into how environments change over time. However, these unified or video-generation-based architectures are often built independently from existing pretrained VLAs, making them difficult to integrate with existing high-performing VLA frameworks. Moreover, while effective for capturing visual consistency, these methods largely overlook motion learning – they learn to predict what the scene will look like, rather than how the robot should move. To address these issues, we propose joint learning with motion image diffusion, a simple yet effective strategy that seamlessly improves VLAs with motion reasoning ability. Specifically, as illustrated in Fig. 1 , we extend VLA architecture with a dual-head design: an action head that predicts action chunks as in vanilla VLAs, and a motion head implemented as a Diffusion Transformer (DiT) 51 that predicts optical-flow-based future motion images through diffusion. Both heads share the same VLM backbone and are optimized jointly, enabling the model to learn temporally coherent and physically grounded representations that support both fine-grained control and motion understanding. We find that optical-flow-based motion images offer an efficient and control-aligned supervision signal in joint learning. Unlike future image prediction or language-based motion descriptions, optical flow directly encodes how the scene moves, making it inherently consistent with action learning. This complementary supervision encourages the model to align physical motion dynamics with robot control, providing dense temporal guidance that improves visuomotor policy learning. Importantly, our strategy integrates seamlessly into existing large-scale VLA models with no additional inference latency, making it practical for real-world robotic deployment. Our key contributions are summarized as follows: • We propose joint learning with motion image diffusion that seamlessly augments VLA models with motion reasoning capabilities, while preserving their real-time inference efficiency. • We present motion image diffusion, implemented using a Diffusion Transformer (DiT), which provides dense pixel-level dynamic supervision that is complementary to sparse action supervision. We show that the optical-flow-based motion images are the most effective representation for joint action–motion learning. • Extensive experiments demonstrate that our approach improves the performance of large-scale VLA models. For example, we enhance \pi -series VLA models to achieve an average success rate of 97.5% on the LIBERO benchmark and 58.0% on the RoboTwin benchmark. Figure 2 : Overview of joint learning VLA with motion image diffusion. 2 Related Work Vision-Language-Action Models. With recent advances of multimodal Large Language Models (MLLM) 43 ; 1 ; 62 ; 30 ; 3 and large-scale robotic datasets 20 ; 50 ; 31 , Vision-Language-Action (VLA) models have become a promising paradigm for generalist robotic manipulation 8 ; 77 ; 73 ; 38 ; 70 ; 2 ; 11 ; 59 ; 46 ; 32 ; 35 ; 68 ; 53 ; 44 ; 56 ; 45 ; 6 ; 28 ; 9 ; 55 ; 57 ; 52 . They are typically initialized from pretrained large models to inherit understanding and generalization abilities. RT series 8 ; 77 ; 2 pioneer the attempt to finetune MLLM for robotic manipulation, showing strong performance and generalization capability. OpenVLA 32 scales VLA with a 7B model trained on 970k real-world demonstrations spanning diverse datasets. \pi series 6 ; 28 adapt the PaliGemma 3 architecture with an action expert via flow matching 41 . Despite the promising trend, these methods primarily focus on learning a direct mapping from language and observations to actions, which lacks explicit motion learning. Motion Learning in VLA Models. Motion learning is a fundamental direction for understanding temporal dynamics in robotic manipulation 23 ; 5 ; 18 ; 27 ; 54 ; 17 ; 40 ; 21 . Recently, several studies have explored incorporating motion as an additional modality into large-scale VLA frameworks 75 ; 74 . For example, FlowVLA 75 introduces optical flow images through visual Chain-of-Thought (CoT) reasoning for world model pre-training. While this enhances motion reasoning, the flow signal is used only during pretraining and is not integrated into the robot policy, limiting its influence on downstream action generation. Similarly, TraceVLA 74 and WorldVLA 10 employ predictive

Read the complete text