Moto: Latent Motion Token as the Bridging Language for Learning Robot Manipulation from Videos Yi Chen, Yuying Ge, Weiliang Tang, Yizhuo Li, Yixiao Ge, Mingyu Ding, Ying Shan, Xihui Liu Abstract Recent developments in Large Language Models (LLMs) pre-trained on extensive corpora have shown significant success in various natural language processing (NLP) tasks with minimal fine-tuning. This success offers new promise for robotics, which has long been constrained by the high cost of action-labeled data. We ask: given the abundant video data containing interaction-related knowledge available as a rich “corpus”, can a similar generative pre-training approach be effectively applied to enhance robot learning? The key challenge is to identify an effective representation for autoregressive pre-training that benefits robot manipulation tasks. Inspired by the way humans learn new skills through observing dynamic environments, we propose that effective robotic learning should emphasize motion-related knowledge, which is closely tied to low-level actions and is hardware-agnostic, facilitating the transfer of learned motions to actual robot actions. To this end, we introduce Moto , which converts video content into latent Mo tion To ken sequences by a Latent Motion Tokenizer, learning a bridging “language” of motion from videos in an unsupervised manner. We pre-train Moto-GPT through motion token autoregression, enabling it to capture diverse visual motion knowledge. After pre-training, Moto-GPT demonstrates the promising ability to produce semantically interpretable motion tokens, predict plausible motion trajectories, and assess trajectory rationality through output likelihood. To transfer learned motion priors to real robot actions, we implement a co-fine-tuning strategy that seamlessly bridges latent motion token prediction and real robot control. Extensive experiments show that the fine-tuned Moto-GPT exhibits superior robustness and efficiency on robot manipulation benchmarks, underscoring its effectiveness in transferring knowledge from video data to downstream visual manipulation tasks. Figure 1 : The overview of Moto , which utilizes Latent Mo tion To kens as a bridging “language” for autoregressive pretraining on video data. The Moto-GPT pre-trained through next motion token prediction learns a wealth of motion-related prior knowledge from videos, which can be seamlessly transferred to enhance downstream robot manipulation tasks with significant performance gains. * * footnotetext: Part of the work done during internship at Tencent ARC Lab. † † footnotetext: Corresponding Authors. 1 Introduction Recent advancements in Natural Language Processing (NLP) have stemmed from successful autoregressive pre-training on large text corpora via next-word prediction [ 45 , 5 , 43 , 50 , 17 ] . Pre-trained Large Language Models (LLMs) have shown exceptional performance across various downstream NLP tasks after fine-tuning on smaller datasets. This success opens new opportunity for robotics, which has been limited by the high costs of action-labeled data. Given the abundance of interaction-rich video data [ 1 , 57 ] , we ask: Can we leverage autoregressive pre-training on video data to improve robot learning? The main challenge is finding an appropriate representation for autoregressive pre-training on video data that effectively captures prior knowledge for robot manipulation. Pioneering research in video pre-training for robotics primarily focused on static frames, emphasizing frame-level visual details [ 54 , 8 , 18 ] . However, humans learn skills by observing dynamic environments, focusing on changes in state—what we term motion. Thus, we argue that effective autoregression for robotics should prioritize motion-related knowledge , which aligns closely with low-level robot actions and is hardware-agnostic, facilitating the transfer of learned motions to actual robot actions through fine-tuning. In this work, we introduce Moto , which utilizes Latent Mo tion To kens as a bridging “language” to model visual motions between video frames in an unsupervised manner. As illustrated in Fig. 1 , we first train a discrete Latent Motion Tokenizer to produce compact latent motion tokens that capture dynamics between video frames without external supervision. We then pre-train Moto-GPT using a GPT-based architecture to predict the next latent motion token, absorbing motion priors from videos. These learned priors are subsequently transferred to enhance robot manipulation performance through a co-fine-tuning strategy. Specifically, as shown in Fig. 2 , the Latent Motion Tokenizer encoder employs a VQ-VAE-based architecture [ 51 ] to compress two successive video frames into discrete tokens. By regularizing the decoder to reconstruct the second frame from the first frame and the tokens, the tokenizer is trained to effectively capture the changes between video frames, which often arise from motion. Once the tokenizer is trained, we obtain latent motion tokens of every two consecutive frames in a video clip and concatenate them into a sequence to represent the motion trajectory. Subsequently, Moto-GPT is pre-trained on these sequences by predicting the next token based on the initial frame and corresponding language instruction. After this pre-training phase, Moto-GPT is capable of generating plausible trajectories by predicting latent motion tokens autoregressively. To adapt Moto-GPT for downstream robot manipulation tasks, we concatenate action query tokens with latent motion token chunk at each time step for co-fine-tuning on action-labeled robot data. The action query tokens are processed by a learnable module to predict low-level actions, while the motion tokens are fine-tuned using the original next-token prediction mechanism. This co-fine-tuning strategy effectively transfers abstract intentions in learned motion priors into precise action execution, allowing the model to utilize the inherent knowledge of the pre-trained Moto-GPT for successful manipulation. We conduct extensive experiments to validate our claims from various perspectives: (1) Latent Motion Tokens as an Interpretable Motion Language : Experiments show that latent motion tokens encapsulate compact and expressive representations of visual motions and exhibit promising cross-embodiment transfer ability (even from human to robot). (2) Pre-trained Moto-GPT as a Useful Motion Prior Learner : Results indicate that pre-trained Moto-GPT achieves promising outcomes in predicting plausible motion trajectories and assessing the rationality of robot trajectories based on output likelihood. (3) Fine-tuned Moto-GPT as an Effective Robot Policy : The fine-tuned Moto-GPT demonstrates significant performance gains over counterparts trained from scratch without motion priors, especially with limited training data. The performance can be further boosted with human video pre-training, highlighting the potential of our approach in transferring motion knowledge learned from Internet-scale videos to robot manipulation. In summary, our contributions are as below: • Introduction of Latent Motion Tokens, which model visual motions between video frames in an unsupervised manner, serving as a bridging “language” for autoregressive pre-training to enhance robot learning. • Pre-training of Moto-GPT through next latent motion token prediction on video data, enabling the model to learn useful motion priors without requiring action annotations. • Implementation of a co-fine-tuning strategy to successfully transfer learned motion priors to actual robot manipulations, with the fine-tuned model showing competitive performance on robotic benchmarks. • We conduct systematic experiments and analyses to validate each training stage of our approach and the effectiveness of the resulting robot policy. Our open-source code provides clear guidance for future exploration. 2 Related Work Figure 2 : Overview of Moto’s three training stages: (1) The Latent Motion Tokenizer encodes key visual motions between video frames into compact latent tokens in an unsupervised manner using pure video data. (2) Moto-GPT is pre-trained with autoregressive motion token prediction to learn motion priors from video-instruction pairs. (3) Moto-GPT is co-fine-tuned on action-labeled trajectories to predict robot actions based on the output of learnable action query tokens while maintaining the next-motion-token prediction objective. Vision-Language-Action Models. Recent studies have utilized transformers as vision-language-action (VLA) architectures to generate robot actions from observations and language instructions [ 4 , 47 , 25 , 30 ] . VLA model pre-training has gained traction, with approaches either fine-tuning policy models from powerful vision-language models pre-trained on large image-text datasets [ 15 , 62 , 32 ] or training generalist models on diverse robot data with action labels [ 41 , 27 , 14 , 52 , 2 ] . Our work enhances VLA models through generative pre-training on video data, providing richer interaction details. Beyond VLA models, several studies improve robot manipulation by incorporating multi-perspective views and depth information [ 35 , 7 , 59 ] , and employing techniques like action chunking and policy diffusion for precision [ 22 , 11 , 26 ] . Additionally, some works [ 19 , 34 ] decompose high-level language instructions into latent skills via auxiliary training objectives. Robot Learning from Videos. Early works [ 42 , 37 ] used contrastive learning with egocentric videos to enhance visual representations. Some studies [ 16 , 28 , 33 , 3 , 29 ] generate videos as intermediate plans for low-level control. Recent research [ 54 , 8 , 23 ] has focused on generative video pre-training for end-to-end policy models. Escontrela et al. [18] pre-train autoregressive video prediction models for reinforcement learning. These works primarily use pixel values or patch-level tokens. In contrast, our approach targets latent motion tokens, emphasizing key visual motions. Some studies build world models through action-conditioned video generation [ 21 , 56 , 55 ] , aiding reinforcement learning. Genie [ 6 ] introduces unsupervised learning of latent actions from large-scale videos for 2D gaming simulators. Similar latent action concepts are explored in LAPO [ 49 ] and DynaMo [ 12 ] , though LAPO is limited to video games and DynaMo focuses on visual representation learning. Concurrently, LAPA [ 58 ] is pre-trained to predict one-step future latent action, while IGOR [ 10 ] uses latent actions as intermediate goals. Our work differs by pre-training an end-to-end policy model to autoregressively predict a trajectory of latent motion tokens for future video clips, providing a more natural modeling of sequential visual motions from videos. Additionally, we conduct a rigorous analyses to validate the effectiveness of policy and each training stage, offering clear guidance for future research. 3 Methodology 3.1 Overview Moto utilizes autoregressive generative pre-training on latent motion token sequences to learn motion priors from videos, followed by co-fine-tuning on action-labeled data for robot control. As illustrated in Fig. 2 , Moto consists of three stages: 1) unsupervised training of the Latent Motion Tokenizer, 2) pre-training of the generative model Moto-GPT, and 3) co-fine-tuning for robot action policy. In Sec. 3.2 , we detail the Latent Motion Tokenizer, which encodes visual dynamics into quantized latent motion tokens. We also describe the training procedures for Moto-GPT, including motion token autoregressive pre-training in Sec. 3.3 and supervised co-fine-tuning in Sec. 3.4 . Implementation details can be found in the Supplementary Material. 3.2 Latent Motion Tokenizer The Latent Motion Tokenizer, as shown in Fig. 2 , learns a latent ‘‘language’’ to capture essential visual motions between successive video frames 1 1 1 To ensure significant visual differences, we down-sample the original video by a certain rate. in an unsupervised manner. The architecture follows a standard auto-encoder design for motion tokenization and detokenization. The tokenization employs an M-Former, a multi-layer transformer that extracts motion features from the last-layer patch features of the current frame o_{t} and the preceding frame o_{t-1} using a frozen pre-trained ViT encoder [ 24 ] . We concatenate 8 learnable query embeddings with these patch features as additional input to the M-Former, where the queries interact through self-attention layers. The output query features are then processed by a VQ codebook with a vocabulary size of 128 to produce discrete latent motion tokens. For de-tokenization, we use a ViT Decoder for image reconstruction, which takes the linearly embedded patches of o_{t-1} and recovers the pixel values for o_{t} based on the latent motion tokens. An MLP projects the concatenated quantized embeddings of the latent motion tokens into a compact embedding (1 token), which is added to each input patch embedding. This conditional embedding acts as an information bottleneck between the encoder and decoder, enabling the ViT Decoder to capture nuanced changes between frames and accurately transform o_{t-1} into o_{t} . The components of the Latent Motion Tokenizer are jointly optimized using the standard VQ-VAE objective [ 51 ] , which includes reconstruction loss, vector quantization loss, and commitment loss. We specifically use the MSE loss between the output pixel values from the ViT Decoder and the ground-truth pixel values of o_{t} as the reconstruction loss. Once trained, the Latent Motion Tokenizer is frozen to produce unified sequential motion representations for videos through “bi-frame” tokenization. Additionally, with the initial observation and specified latent motion tokens, the decoder can function as a “simulator” to generate rollouts for visualizing environmental changes. 3.3 Motion Token Autoregressive Pre-training With the Latent Motion Tokenizer, Moto-GPT is allowed to learn about diverse visual motions from videos, using latent motion tokens as a bridging language. As shown in Fig. 2 , Moto-GPT is pre-trained with a next-motion-token prediction objective. For a video clip [o_{0},o_{1},...,o_{T}] , we derive a chunk of latent motion tokens for each pair of consecutive frames, concatenating them chronologically to form a sequence. Moto-GPT employs a GPT-style transformer for autoregression on these motion token trajectories. Additionally, we prepend the text features from the instruction and the visual features from the initial video frame as input prompts. The pre-training objective maximizes the likelihood of the ground-truth latent motion token sequence given the language instruction and the initial video frame: \mathcal{L}_{motion}=-\sum_{i=1}^{M}\log P(m_{i}|\boldsymbol{l},\boldsymbol{v},\boldsymbol{m}_{