Learning Action Priors for Cross-embodiment Robot Manipulation
Dong Jing, Tianqi Zhang, Jiaqi Liu, Jinman Zhao, Zelong Sun, Li Erran Li, Zhiwu Lu, Mingyu Ding, [0.20em] Renmin University of China, University of North Carolina at Chapel Hill, University of Toronto, Amazon · 2026
Read Learning Action Priors for Cross-embodiment Robot Manipulation by Dong Jing, Tianqi Zhang, Jiaqi Liu, Jinman Zhao, Zelong Sun, Li Erran Li, Zhiwu Lu, Mingyu Ding, [0.20em] Renmin University of China, University of North Carolina at Chapel Hill, University of Toronto, Amazon (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26095.txt.
Introduction
Dong Jing leads a team teaching robot action modules some physics before vision and language enter. Vision-language-action models graft an action head onto a vision-language backbone. The backbone brings strong sight and language priors. Motion, though, must be learned nearly from scratch. Early training then juggles two hard jobs at once: action timing and cross-modal alignment. Different robot bodies, with different action patterns, make the juggling harder.
Pretraining the action module is the fix. Stage one trains a light flow-matching encoder-decoder on raw action trajectories alone. The module absorbs cross-embodiment motion structure first. Stage two transfers that prior into full model training. The decoder is reused, and latent distillation aligns sight and language features early. The encoder doubles as a history compressor, squeezing past states and actions into one context token. Thirteen tasks across simulated and real platforms validate the approach.
Catalog record
| CREATOR | Dong Jing, Tianqi Zhang, Jiaqi Liu, Jinman Zhao, Zelong Sun, Li Erran Li, Zhiwu Lu, Mingyu Ding, [0.20em] Renmin University of China, University of North Carolina at Chapel Hill, University of Toronto, Amazon |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.26095
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Action Priors for Cross-embodiment Robot Manipulation Dong Jing, Tianqi Zhang, Jiaqi Liu, Jinman Zhao, Zelong Sun, Li Erran Li, Zhiwu Lu, Mingyu Ding, [0.20em] Renmin University of China, University of North Carolina at Chapel Hill, University of Toronto, Amazon Abstract Most Vision-Language-Action (VLA) models build on a Vision-Language Model (VLM) backbone by attaching an action module and optimizing the full policy jointly. This design inherits strong visual and linguistic priors from the VLM, but leaves the action module to learn physical motion almost from scratch. As a result, the policy lacks an explicit motion prior, forcing early optimization to simultaneously discover temporal action dynamics and cross-modal alignment, a challenge further amplified in cross-embodiment settings with heterogeneous action distributions. In this work, we propose to pretrain the action module with motion priors before cross-modal VLA alignment. Specifically, we introduce a two-stage training framework that equips the action module with cross-embodiment temporal motion structure before VLA training begins. In Stage 1, a lightweight flow-matching-based encoder-decoder action module efficiently learns temporal motion structure solely from unconditioned action trajectories, without processing visual or language tokens. In Stage 2, this learned prior is transferred to VLA training through decoder reuse and early-stage latent distillation, aligning visual-language features with the action embedding space while still allowing end-to-end policy refinement. In addition, the trained encoder serves as a compact history compressor, summarizing state-action histories into a single temporal context token for history-aware modeling at negligible cost. Extensive experiments across 13 diverse cross-embodiment tasks on both simulated and real-world platforms validate the effectiveness of our approach. Compared with VLA training without action priors, our model achieves faster convergence, higher success rates, and substantially stronger performance on data-scarce real-world tasks. Moreover, scaling up the action data in Stage 1 yields a more generalizable action prior that directly improves downstream VLA performance. Fig. 1 : Illustration of motivation: a policy should first learn to move, and then learn to see and act. (a) In Stage 1, the action module is trained purely on action trajectories, without any visual observation or language instruction, to efficiently acquire a general action prior. (b) In Stage 2, the prior is transferred to VLA training, enabling model to perceive scenes and follow instructions with an action module that already knows how to move. With the action prior, the policy produces smooth trajectories. Without it, the trajectories are unstable. (c) The action prior accelerates VLA training and improves performance. I Introduction In recent years, VLAs have rapidly emerged as a dominant paradigm for robotic manipulation. These models build upon foundation VLMs [ 44 , 19 , 5 , 1 ] or video generation models [ 25 , 8 , 14 , 2 , 68 ] , which encode rich multi-modal semantic knowledge. Such semantic foundations allow VLAs to connect high-level language instruction and visual observations with low-level physical execution. This paradigm has achieved remarkable success across diverse robotic platforms, such as tabletop manipulators [ 10 , 9 , 39 ] , quadruped robots [ 47 , 31 ] , and humanoid systems [ 26 , 6 , 15 ] . These advances open a credible path toward generalist robotic policy. However, the success of current VLAs is still largely driven by visual and linguistic priors inherited from large-scale foundation-model training, while the action generation module itself usually receives no comparable prior over physical motion. Missing action priors create a key imbalance in the standard VLA training pipeline. Existing methods typically append an action module to a foundation VLM and optimize the whole model jointly via imitation learning [ 9 , 7 , 42 ] . While the VLM already encodes rich semantic structure, the action module is often initialized from random weights or from parameters transferred from unrelated modalities. It therefore leaves the action module to learn physical motion and temporal dynamics from scratch, creating a bottleneck in modeling robot action distributions. As a result, early VLA training must learn both the action distribution and the cross-modal alignment between perception, language, and control at the same time. The gradients produced by such an under-trained action head can be unstable and poorly structured, interfering with the foundation backbone and slowing convergence. This issue becomes more pronounced in cross-embodiment settings [ 50 , 63 ] , where different robots exhibit substantially different tasks, action spaces, and motion distributions. To address this imbalance, we introduce action prior learning before cross-modal VLA alignment , targeting only the action module and using only action data. As shown in Figure 1 (a), the action module first learns to move from unconditioned action trajectories, without any visual observation or language instruction. Much like a blindfolded apprentice practicing basic movements, it focuses on the intrinsic temporal structure of physical motion instead of learning action dynamics and cross-modal alignment simultaneously. The learned prior is then transferred to Stage 2, where the VLA learns to perceive the scene and follow instructions on top of an action module that already encodes plausible motion patterns, as illustrated in Figure 1 (b). This decoupled training process yields smoother trajectories and faster convergence, as summarized in Figure 1 (b,c), because VLA training begins from a structured action module rather than an untrained action head. We instantiate this action prior with an encoder-decoder action module. The encoder is designed to extract the motion semantics that emerge across an action sequence, rather than from isolated atomic actions. A single action usually records only an infinitesimal state change over a short time interval, while meaningful behaviors such as translating, rotating, or grasping arise from the continuous composition of actions over a temporal horizon. The encoder therefore compresses an entire action chunk into a compact latent embedding that captures its temporal dynamics and macroscopic motion structure. Conditioned on this embedding, the decoder reconstructs the original action chunk via a flow-matching objective, which provides a flexible way to model the continuous action distribution and its uncertainty. We transfer this learned prior to VLA training through three tightly connected uses of the learned action module: decoder reuse , early-stage latent distillation , and history compression . At the action-output side, the decoder that already models plausible robot motion from Stage 1 is reused as the VLA action head. At the representation side, the encoder provides a latent target for early-stage distillation: given the ground-truth action chunk, it produces a structured embedding that supervises the VLM’s predicted action feature and quickly anchors it to the learned motion latent space. This alignment is gradually relaxed, allowing later training to refine the policy end-to-end without being permanently constrained by the encoder. Finally, the encoder also serves as a compact history compressor, mapping past state-action trajectories into a single latent token that injects temporal context into the VLM. Together, these mechanisms turn the action prior from a standalone reconstruction model into a practical initialization, alignment signal, and memory interface for VLA training, leading to smoother rollouts and faster convergence as shown in Figure 1 (b,c). We evaluate our method on a demanding cross-embodiment setup that spans 13 tasks across two simulated benchmarks, LIBERO and RoboCasa, and a real-world Franka platform. All models are trained jointly on heterogeneous embodiments and evaluated without environment-specific fine-tuning. Our model has several appealing benefits.