Future Optical Flow Prediction Improves Robot Control & Video Generation
Kanchana Ranasinghe, Honglu Zhou, Yu Fang, Luyu Yang, Le Xue, Ran Xu, Caiming Xiong, Silvio Savarese, Michael S Ryoo, Juan Carlos Niebles · 2026
Read Future Optical Flow Prediction Improves Robot Control & Video Generation by Kanchana Ranasinghe, Honglu Zhou, Yu Fang, Luyu Yang, Le Xue, Ran Xu, Caiming Xiong, Silvio Savarese, Michael S Ryoo, Juan Carlos Niebles (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10781.txt.
Introduction
Kanchana Ranasinghe and co-authors present FOFPred, a model that forecasts future optical flow. Optical flow maps how each point in a scene is about to move. Such maps are gold for robot control and for video generation. Yet forecasting dense motion that holds up in the real world is still an open challenge.
FOFPred is conditioned on language, so words can steer the forecast. It joins a vision-language model with a diffusion process in one design. Learning comes from noisy, real-world data, not just clean lab sets. A robot that can foresee motion can plan smoother paths, and a video model that foresees flow can keep motion coherent across frames.
Catalog record
| CREATOR | Kanchana Ranasinghe, Honglu Zhou, Yu Fang, Luyu Yang, Le Xue, Ran Xu, Caiming Xiong, Silvio Savarese, Michael S Ryoo, Juan Carlos Niebles |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-10781
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Future Optical Flow Prediction Improves Robot Control & Video Generation Kanchana Ranasinghe, Honglu Zhou, Yu Fang, Luyu Yang, Le Xue, Ran Xu, Caiming Xiong, Silvio Savarese, Michael S Ryoo, Juan Carlos Niebles Abstract Future motion representations, such as optical flow, offer immense value for control and generative tasks. However, forecasting generalizable spatially dense motion representations remains a key challenge, and learning such forecasting from noisy, real-world data remains relatively unexplored. We introduce FOFPred, a novel language-conditioned optical flow forecasting model featuring a unified Vision-Language Model (VLM) and Diffusion architecture. This unique combination enables strong multimodal reasoning with pixel-level generative fidelity for future motion prediction. Our model is trained on web-scale human activity data—a highly scalable but unstructured source. To extract meaningful signals from this noisy video-caption data, we employ crucial data preprocessing techniques and our unified architecture with strong image pretraining. The resulting trained model is then extended to tackle two distinct downstream tasks in control and generation. Evaluations across robotic manipulation and video generation under language-driven settings establish the cross-domain versatility of FOFPred, confirming the value of a unified VLM-Diffusion architecture and scalable learning from diverse web data for future optical flow prediction. Figure 1 : Our proposed model, FOFPred, learns generalizable language-driven future optical flow prediction that supports robot control and video generation. <\cdot> is a placeholder for text (e.g., Left/Up/Right). For visualization purposes, we sample points from the predicted future optical flow and show their predicted trajectories in this figure. Checkout our website for more visualizations. ⋆ Work done during an internship at Salesforce. 1 Introduction Recent studies have investigated the use of motion representations such as optical flow Xu et al. 2024 ; Ranasinghe et al. 2025 , sparse motion trajectories Yuan et al. 2024 ; Wen et al. 2023 ; Bharadhwaj et al. 2024 , and direction commands Burgert et al. 2025 ; Wang et al. 2024b ; Yin et al. 2023 for multiple control and generation tasks. The main idea behind such motion representations, often in the form of future pixel movements like optical flows and trajectories, is to explicitly capture desired dynamics necessary for downstream tasks such as robot control and video generation. In the case of robot control, models with such future motion representations have the capability to explicitly consider future pixel displacement information to infer robot actions Yuan et al. 2024 ; Wen et al. 2023 ; Xu et al. 2024 ; Bharadhwaj et al. 2024 ; Ranasinghe et al. 2025 . This is in contrast to the conventional Vision-Language-Action (VLA) models taking only RGB-frames and language instructions as input. Similarly, video diffusion models utilizing motion representations enables generation of videos with consistent and detailed movements, and it does it better than the counterparts utilizing language-only instructions Chefer et al. 2025 ; Liang et al. 2024 ; Koroglu et al. 2024 ; Gao et al. 2025 . The key is in reliable and generalizable computation of such motion representation, which often involves future forecasting . What has to be provided to the models for many (robot) control and (video) generation tasks is how things should move in the future, not what motion has been observed in the past. This inherently is a challenging task: it requires a clear formulation of what type of ‘future motion’ we will be computing, curating large enough training data with strong supervisory signals, and an elegant model capable of fully digesting such (potentially cross-domain) training data while generalizing across domains. In this paper, we formulate the problem as the forecasting of future optical flows conditioned on language instructions, and present a new model architecture to forecast them. We also discuss how we can train our model with videos from various sources, ranging from human videos to robot videos. Optical flows are spatially dense motion representations, capturing movements of every pixel in the scene. This better preserves motion details compared to sparse representation, and enables easy use of diffusion models to learn their distributions in the form of images. In contrast to predicting RGB frame sequences, which also capture future motion Hu et al. 2025 , optical flow eliminates static information irrelevant to motion, leading to more motion-centric representations Ranasinghe et al. 2025 . We introduce an optical flow forecasting model unifying both Vision-Language Models (VLM) and Diffusion. Fully taking advantage of VLM reasoning capability and pre-trained diffusion model image generation ability, we train our VLM-Diffusion model to generate future optical flow images. Our model is trained on web-scale human activity videos with paired captions allowing highly-scalable training. The resulting model, named FOFPred, enables highly generalizable language-driven future optical flow prediction as illustrated in Figure 1 . While learning from web videos provides this strong generality, such data is highly unstructured with noisy videos and captions. Most prior work learning future motion representations either avoid training with such data Yuan et al. 2024 ; Wen et al. 2023 ; Xu et al. 2024 ; Bharadhwaj et al. 2024 ; Ranasinghe et al. 2025 or utilize RGB frame prediction Hu et al. 2025 ; Ko et al. 2023 . In Yang et al. 2025 where such training data is used, explicit camera motion correction is used to handle noisy videos and a VLM backbone enables learning from diverse noisy captions. However, the use of only a VLM restricts their future motion prediction to sparse pixel trajectories. In our work, we calculate dense relative optical flow compensating for the camera motion and model this with diffusion, which improves our learning from noisy web videos. Finally, we validate the effectiveness and generality of FOFPred through evaluation on two challenging downstream tasks: language-driven robotic manipulation and language-guided motion video generation. For each task, we attach a diffusion policy head or a video diffusion head on top of FOFPred and fine-tune the two heads separately for the respective downstream domains. We summarize our key contributions as follows: • Unified VLM-Diffusion: Adopting this recent architecture for generalizable future optical flow prediction. • Scalable Learning: Establish framework for learning future optical flow prediction from web-scale human activity videos. • Cross-Domain Versatility: To the best of our knowledge, the first to utilize VLM–diffusion backbone for robotic manipulation and controlled video generation under language-driven settings. Figure 2 : Overview of Proposed FOFPred: (Left & Center) We present the unified VLM-Diffusion architecture used in FOFPred. Only the DiT module is trained while the VAE and VLM remain frozen. (Right) We illustrate two distinct pipelines constructed with FOFPred for two orthogonal tasks in control and generation. Each task specific head is first finetuned prior to inference on the downstream task. 2 Related Work Unified Model for Control Generation. Recent progress in foundation models has demonstrated a powerful trend toward unified model architectures that integrate diverse modalities and achieve highly controlled visual input manipulation. These generalist models Nvidia et al. 2025 ; Bao et al. 2023 ; Chen et al. 2025a ; Yang et al. 2025 ; Xiao et al. 2024 ; Wu et al. 2025 ; Pan et al. 2025 ; Ju et al. 2025 ; Gupta et al. 2025 , often building on transformers and large-scale pre-training (e.g., Gato Reed et al. 2022 ), establish single frameworks capable of handling hundreds of tasks. In the generative domain,