Autoregressive Flow Matching for Motion Prediction

Johnathan Xie, Stefan Stojanov, Cristobal Eyzaguirre, Daniel L. K. Yamins, Jiajun Wu · 2025

Read Autoregressive Flow Matching for Motion Prediction by Johnathan Xie, Stefan Stojanov, Cristobal Eyzaguirre, Daniel L. K. Yamins, Jiajun Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22688.txt.

Introduction

Johnathan Xie and five co-authors present Autoregressive Flow Matching, a probabilistic method for predicting how points move through a scene. Motion prediction models usually train on narrow data for one job, while giant video predictors look realistic yet fumble complex motion despite their scale. Trained on diverse video datasets, their model forecasts future point track locations over long horizons, treating motion itself as the object to model.

The team built benchmarks for human and robot motion prediction to judge the approach. Predicted tracks proved useful as conditioning: feeding a robot action predictor or a human motion predictor these forecasted tracks greatly improved the downstream tasks. Code and trained models are publicly available, and the authors pitch the method as general infrastructure for sequential continuous data.

Catalog record

CREATORJohnathan Xie, Stefan Stojanov, Cristobal Eyzaguirre, Daniel L. K. Yamins, Jiajun Wu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22688

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Autoregressive Flow Matching for Motion Prediction Johnathan Xie, Stefan Stojanov, Cristobal Eyzaguirre, Daniel L. K. Yamins, Jiajun Wu Abstract Motion prediction has been studied in different contexts with models trained on narrow distributions and applied to downstream tasks in human motion prediction and robotics. Simultaneously, recent efforts in scaling video prediction have demonstrated impressive visual realism, yet they struggle to accurately model complex motions despite massive scale. Inspired by the scaling of video generation, we develop autoregressive flow matching (ARFM), a new method for probabilistic modeling of sequential continuous data and train it on diverse video datasets to generate future point track locations over long horizons. To evaluate our model, we develop benchmarks for evaluating the ability of motion prediction models to predict human and robot motion. Our model is able to predict complex motions, and we demonstrate that conditioning robot action prediction and human motion prediction on predicted future tracks can significantly improve downstream task performance. Code and models publicly available at: https://github.com/Johnathan-Xie/arfm-motion-prediction . 1 Introduction Consider the video frame in the top left of Figure 1 where we see a person standing above a slackline. One plausible future motion would be for the man to fall down then bounce back up on the slackline, or miss and fall to the ground. However, it would not be plausible for the person to fall through the ground or rise above the frame. While humans can easily understand plausible future motion given just a single video frame, developing a method to probabilistically model future motion is challenging as it requires understanding agent behavior and physics. In this paper, we aim to predict future motion with vision-language conditioning on a wide range of video domains. Due to the challenging nature of motion prediction, prior works focus on narrow domains such as human motion 43 ; 23 ; 8 , self-driving 54 ; 28 ; 32 , and robotics 4 ; 46 ; 51 ; 53 . While these are successful in their respective domains, they require domain-specific knowledge and lack generality. On the other hand, in recent advances in video generation 41 ; 33 , large-scale training on internet datasets has unlocked the ability to generate visually plausible and diverse videos. Yet, these models often struggle to accurately model motion and realistic physics, and remain prohibitively expensive due to pixel-level synthesis. We take inspiration from both lines of work and explore generalized motion prediction, a formulation that combines the scalability and flexibility of large-scale video models with the efficiency and structure of motion-centric prediction. Unlike prior motion prediction works which focus on a specific domain and video prediction works which focus on visual realism, our proposed formulation aims to predict future motion for a wide range of videos. The task of predicting track motion presents a unique challenge of modeling a multivariate causal time series with vision and language conditioning. Furthermore, since future motion has irreducible uncertainty, there are many future plausible motions when provided few conditioning frames, we must model the future point tracks probabilistically. Lastly, at inference time, we require the ability to efficiently roll out predictions and quickly update them in response to newly provided frames while remaining temporally consistent. To this end, we propose autoregressive flow matching, a new method that combines the strengths of autoregressive modeling with flow matching prediction to probabilistically model future motion. We train our model using pseudo-labels from an off-the-shelf point tracker, which enables our method to easily scale to a large number of diverse video datasets. When trained on these datasets, our motion predictor is capable of producing accurate predictions conditioned on as few as a single frame. Furthermore, we demonstrate the utility of our model by using predicted tracks from our model to condition robotic action prediction and human interaction synthesis, which results in significant improvements in task performance. Our main contributions are: 1. Autoregressive flow matching (ARFM), a new method for efficient probabilistic modeling of sequential continuous data. 2. Methods for predicting future point tracks, updating a constant track horizon in an online fashion, predicting query point information, and encoding point tracks to features, which improve task performance in robotics and interaction synthesis. 3. Track motion prediction benchmarks and training datasets developed using a pseudo-labeling approach to train and evaluate future point track prediction. Figure 1 : Generalized motion prediction via autoregressive flow matching . To train our method (top), we extract pseudo label tracks using an off-the-shelf point tracker, then train an autoregressive flow matching model to predict tracks. To apply our method to downstream tasks (bottom), we generate future point tracks conditioned on provided video frames and an optional language prompt. Then, we extract features from these tracks which are used by a task specific model to accomplish the task. 2 Related Work Optical flow and point tracking. Optical flow is a classic computer vision problem which involves estimating pixel motion between two video frames. A related task is that of point tracking which is the task of predicting point pixel motion and visibility over many frames. Recently, various works ( 39 ; 14 ; 55 ) have improved performance on these two problems through the use of deep learning. Similarly, other works ( 45 ; 9 ) have improved the tracking of points over longer durations and through occlusions. With these recent advancements, we are able to use existing point trackers to pseudo-label videos for training our track prediction model. Video generation Video generation ( 47 ; 13 ; 15 ; 12 ) is closely related to our proposed motion generation task and involves predicting future video frames given past video frames. Previously, video generation has been used to provide a dynamics model for robotics trajectory planning 11 ; 12 , and recent advancements in video synthesis allow for the generation of visually plausible videos from natural language instructions. While these video generation models create visually appealing videos, they still struggle to accurately model complex motion and are incredibly expensive at inference time due to their massive scale. In comparison, our proposed problem seeks to only model the motion aspects of video and can run efficiently by predicting only sparse point tracks. Motion prediction. Prior works have explored the problem of motion prediction in different contexts. In robotics ( 4 ; 51 ; 46 ; 53 ) , recent works have learned conditional motion prediction methods for the purpose of predicting plausible trajectories or learning affordances ( 3 ) . Other works ( 23 ; 43 ; 6 ; 37 ) have trained models to predict short-term object motion on human centric datasets to learn priors for human action recognition and human motion prediction. Many works ( 54 ; 28 ; 32 ) have also studied motion prediction in the context of autonomous driving in order to predict the future occupancy locations of cars in lidar and graph modalities. Compared to these prior works, we perform motion prediction in a generalized fashion with a single foundation model trained on a diverse set of videos enabling zero-shot predictions. 3 Approach In this section, we present our approach to the various sub-problems of generalized motion prediction. We first discuss our pseudo-labeling data pipeline which enables us to train on

Read the complete text