PointAction: 3D Points as Universal Action Representations for Robot Control
Mutian Tong, Han Jiang, Qiao Feng, Lingjie Liu, Jiatao Gu · 2026
Read PointAction: 3D Points as Universal Action Representations for Robot Control by Mutian Tong, Han Jiang, Qiao Feng, Lingjie Liu, Jiatao Gu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03943.txt.
Introduction
Tong and co-authors seek a better bridge from video forecasts to robot actions. Video models know how scenes move, but plain color frames lack depth. They leave contact shape and exact 3D motion unclear. Action labels across tasks and robot bodies are also costly to gather. A shared action form could ease both gaps.
PointAction trains a video model to predict frames plus moving 3D point maps. Those maps track the scene parts that matter for the task. A diffusion decoder then turns point motion into robot commands. The team reports top 4D quality on robot scenes. It beat prior methods in simulation tests. It also transferred to two real robot arms not seen in pretraining.
Catalog record
| CREATOR | Mutian Tong, Han Jiang, Qiao Feng, Lingjie Liu, Jiatao Gu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03943
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PointAction: 3D Points as Universal Action Representations for Robot Control Mutian Tong, Han Jiang, Qiao Feng, Lingjie Liu, Jiatao Gu Abstract Video-Action Models (VAMs) leverage the broad visual dynamics captured by pre-trained video diffusion models, offering a promising path toward generalizable robot manipulation. However, RGB-only video rollouts are not directly actionable: they leave metric 3D motion, contact geometry, and fine-grained spatial constraints under-specified, making action grounding ambiguous. Meanwhile, scaling action supervision across diverse tasks and embodiments remains costly. We present PointAction , a framework that bridges video predictions to robot actions through explicit point-based 4D modeling. PointAction fine-tunes a foundation video generation model to jointly predict future RGB frames and dynamic 3D pointmaps, producing temporally consistent 3D motion of task-relevant scene geometry. These point dynamics serve as a structured, embodiment-agnostic action interface, which a diffusion-based action decoder maps to executable robot actions. By using metric 3D point dynamics as the interface between video prediction and control, PointAction reduces the ambiguity of RGB-only action grounding and supports transfer across tasks and embodiments with limited action supervision. Experiments show that PointAction achieves state-of-the-art 4D generation quality on robot scenes, outperforms existing baselines in simulation, and generalizes to two real robot arms unseen during pretraining. \contribution [†]Equal contribution \correspondence \metadata [ \bsicon Project page: ] https://oriontmt.github.io/pointaction/ 0 0 footnotetext: Work done during internship at University of Pennsylvania. Figure 1 : PointAction enables generalizable robot control via explicit 4D point modeling. From an input image and instruction, our framework jointly synthesizes consistent RGB frames and dynamic, pixel-aligned 3D pointmaps (XYZ). These dynamic 3D robot points serve as a generalizable and scalable proxy for action decoding, seamlessly bridging 2D video priors to robust visuomotor control. 1 Introduction Building generalist robot policies capable of solving diverse manipulation tasks across environments and embodiments remains a central goal in robot learning. Recent Vision-Language-Action (VLA) models [ 27 , 69 , 73 , 30 , 8 , 7 , 6 ] have made remarkable progress by combining the semantic priors of large pre-trained Vision-Language Models (VLMs) [ 51 , 3 ] with large-scale robot datasets [ 40 , 28 , 53 ] . These models can follow language instructions and recognize task-relevant objects with increasing robustness. However, semantic visual understanding alone does not provide an explicit model of how scenes evolve through contact, motion, and long-horizon interaction. As a result, policies trained primarily from paired observation–action data can struggle when manipulation requires reasoning about future physical changes beyond the training distribution. To incorporate such dynamics, recent work has explored Video-Action Models (VAMs) [ 18 , 25 , 9 , 41 , 31 , 29 ] , which use pretrained video diffusion backbones [ 1 , 54 , 61 , 58 ] to predict future scene rollouts as an explicit reasoning trace for action generation. Existing methods either jointly model future observations and actions in an end-to-end process [ 31 , 62 ] , or first predict future observations and then decode actions through an inverse-dynamics module [ 18 , 25 , 9 , 41 ] . Despite their promise, current VAMs still face two tightly coupled obstacles. First, even when actions are predicted jointly or decoded from rollouts, the intermediate representation is typically RGB-dominant: 3D motion, contact-relevant geometry, and fine-grained spatial constraints remain implicit, forcing the action module to learn a difficult mapping from appearance changes to controls. Second, learning this implicit grounding requires substantial paired observation–action supervision [ 20 ] ; however, such data is costly to collect, tied to specific robot embodiments, and difficult to scale across diverse tasks and environments. We argue that dynamic 3D pointmaps provide a natural way to break this representation–supervision bottleneck. As an intermediate representation, they make predicted rollouts more actionable by exposing how task-relevant 3D geometry moves over time, including metric motion and contact-relevant spatial constraints that RGB frames leave implicit. As a training signal, point-level supervision can be obtained at scale from videos through multi-view reconstruction, monocular depth estimation, and motion cues, allowing the rollout model to learn broad scene dynamics beyond robot-specific action labels. Building on this insight, we propose PointAction, a framework that connects video prediction to robot control through explicit point-based 4D modeling. Concretely, PointAction factorizes video-to-action learning into two components: (1) a universal video-to-point model that predicts RGB rollouts together with dynamic 3D pointmaps, and can be pretrained on large-scale video data across tasks and embodiments; and (2) an embodiment-specific point-to-action decoder that maps predicted point dynamics to executable controls and can be adapted with limited robot data. We implement the universal component by fine-tuning a foundation robot video model [ 16 ] to jointly generate RGB frames and spatially aligned XYZ pointmaps, and train a diffusion-based action decoder for action translation. Comprehensive evaluations show that PointAction achieves state-of-the-art 4D generation quality on robot scenes, outperforms existing VLA and VAM baselines in simulation, and transfers to real robot embodiments unseen during pretraining. Our main contributions are threefold: • Point-based interface for video-to-action learning. We introduce PointAction, which uses dynamic 3D pointmaps as an explicit interface between video prediction and robot control, reducing RGB-only grounding ambiguity. • Scalable pretraining with embodiment-specific decoding. PointAction separates a universal video-to-point model, pretrainable from large-scale video data, from a lightweight point-to-action decoder that adapts to each robot with limited action supervision. • Strong 4D generation and manipulation results. Experiments show state-of-the-art 4D generation quality on robot scenes, improved simulation performance over VLA and VAM baselines, and transfer to real robot embodiments unseen during pretraining. 2 Related Works Vision-Language-Action Models. Vision-Language-Action (VLA) models [ 27 , 69 ] achieve strong manipulation performance by leveraging pretrained vision-language backbones and fine-tuning on large-scale robot datasets [ 40 , 28 , 53 ] . RT-2 [ 73 ] co-trained VLMs on robotic trajectories and web-scale vision-language data, and subsequent work [ 30 , 39 , 8 , 7 , 6 ] improves performance via larger datasets and better architectures. A key gap is the insufficient modeling of world dynamics. Recent VLAs incorporate visual-guided action prediction via intermediate future-image rollouts: CoT-VLA [ 66 ] autoregressively predicts subgoal images as explicit visual CoT; UniVLA [ 55 ] and GR-series [ 60 , 14 ] pretrain on future images; WorldVLA [ 13 ] and RynnVLA-002 [ 12 ] jointly optimize future image generation and action prediction; DreamVLA and 3DVLA [ 67 ] further incorporate additional modalities such as depth. In contrast to these VLM-centric approaches, we build on a pretrained video diffusion model [ 16 ] to better capture temporal and fine-grained dynamics. Video-Action Models. Video-Action Models (VAMs) leverage pretrained video generators for robot control and fall into two paradigms. Decoupled VAMs [ 18 , 25 , 9 , 23 , 33 , 41 , 31 , 1 , 54 ] fine-tune a video model on robotic data and decode actions via an inverse-dynamics module, with VPP [ 23 ] and VideoPolicy [ 33 ] extracting actions from the video latent for robustness. End-to-end VAMs [ 21 , 32 , 70 , 49 , 61 , 29 ] jointly generate future frames and actions in a single process, e.g., Cosmos-Policy [ 29 ] encodes robot state, action, and value as pseudo video frames. 4D Modeling for Robot Control.