Supervise What Survives: Geometry-Guided VLA Adaptation from Synthetic Robot Videos
Danze Chen Yanzhe Chen Qiming Huang Zhijun Cao, [0.1em] Chen Gao Mike Zheng Shou [0.4em] Show Lab, National University of Singapore · 2026
Read Supervise What Survives: Geometry-Guided VLA Adaptation from Synthetic Robot Videos by Danze Chen Yanzhe Chen Qiming Huang Zhijun Cao, [0.1em] Chen Gao Mike Zheng Shou [0.4em] Show Lab, National University of Singapore (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24448.txt.
Introduction
Chen and colleagues argue that synthetic robot videos should teach a robot's eyes, never its hands. Vision-language-action models need huge numbers of video action pairs. Real remote control data is scarce, so generated robot videos look like a scalable substitute. Earlier approaches infer approximate actions from the synthetic frames. The team contends in 2026 that this is a mismatch.
A video captures geometry, the spatial path of the task. A real demonstration instead records control in full, down to the precise motor commands. Their Asymmetric Preservation Principle observes that human to robot video generation preserves the two unequally. Visible geometry survives generation. Underlying control signals do not. Their method derives upcoming two-dimensional end effector waypoints from the human recording, after pose estimation, retargeting, simulation, and projection. That surviving geometry teaches visual perception alone. Real demonstrations remain in charge of control. Chen's rule is crisp: supervise only what survives.
Catalog record
| CREATOR | Danze Chen Yanzhe Chen Qiming Huang Zhijun Cao, [0.1em] Chen Gao Mike Zheng Shou [0.4em] Show Lab, National University of Singapore |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24448
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Supervise What Survives: Geometry-Guided VLA Adaptation from Synthetic Robot Videos Danze Chen Yanzhe Chen Qiming Huang Zhijun Cao, [0.1em] Chen Gao Mike Zheng Shou [0.4em] Show Lab, National University of Singapore Abstract Vision-Language-Action (VLA) models require large-scale video-action pairs, yet real teleoperation remains scarce. While generated robot videos offer a scalable alternative, existing methods treat them as real robot data by recovering pseudo-actions from synthesized pixels. We argue that deriving low-level control from generated visuals is a mismatched abstraction. A video captures only geometry : the spatial trajectory representing the where of a task. A real demonstration captures control : the exact motor commands representing the how . Human-to-robot video generation preserves these unequally: the visible geometry survives the generation process, while the underlying control signals are lost. This Asymmetric Preservation Principle dictates a clean rule: this surviving geometry should solely supervise visual perception, leaving control to real demonstrations. Following this principle, we propose GRA ( G eometry-guided R epresentation A lignment), which extracts the geometric content as future 2D end-effector waypoints, computed from the source human video through pose estimation, retargeting, simulation, and calibrated projection, and routes them to the VLA vision backbone via an auxiliary 2D head. The action head is trained on real demonstrations only. During fine-tuning, the waypoint loss persists as a spatial representation anchor that prevents the backbone from losing its geometric grounding. On real-robot tasks, GRA outperforms pseudo-action baselines under matched data budgets and narrows the gap to policies trained with substantially more real demonstrations, suggesting that correctly routed geometry bridges generated videos to robot policies more reliably than recovered actions. 2 2 footnotetext: Corresponding author: [email protected] Keywords: Vision-Language-Action Models, Robot Manipulation, Synthetic Robot Videos, Geometry-Guided Representation Alignment 1 Introduction Vision-Language-Action (VLA) models [ 18 , 4 , 12 , 46 ] unify perception and control but face a severe data bottleneck due to the scarcity of paired video-action data [ 16 , 29 ] : real teleoperation requires dedicated hardware, trained operators, and substantial collection time. Human manipulation videos are abundant, and recent human-to-robot video generation methods [ 36 , 21 , 22 ] make it increasingly feasible to synthesize robot execution videos at scale. However, while these models synthesize visual sequences, they cannot synthesize physical actions. This missing half of the data equation forces us to re-evaluate the exact information these synthetic videos contain, raising a critical question: what supervision can purely visual generated videos reliably provide, and where should it enter the VLA? Figure 1: Illustration of Motivation. (a) Human-to-robot generation produces robot videos at scale, but the generated videos lack action labels. (b) Existing methods recover pseudo-actions from the generated pixels and feed them to the action head, inheriting distorted control signals. (c) GRA routes future 2D end-effector waypoints, extracted from the source human video, to the vision backbone, while the action head trains on real robot demonstrations alone. Existing methods answer this by treating generated videos as demonstrations missing their action labels. Prior work recovers pseudo-actions from synthesized frames using inverse dynamics [ 13 ] or retargeting [ 22 ] , feeding the resulting pairs into the VLA’s action head [ 3 ] . We argue this creates a fundamental video-action mismatch . Video generation optimizes for visual plausibility, not physical control laws. Forcing the action head to map visually hallucinated pixels to rigid real-world control signals conflates features with physical execution, inevitably degrading policy performance. To resolve this mismatch, we must analyze the generation process itself. A video captures only geometry , namely the spatial structure (end-effector trajectories, object arrangements, and coarse motion directions) representing where manipulation unfolds. A real demonstration captures control , which consists of the exact motor commands representing how to execute it. Video generation is fundamentally a visual synthesis process. It captures the appearance of manipulation, not the physics that drives it. The human-to-robot generation process therefore transfers visible geometric structure reliably, but loses the underlying control signals. We formalize this as the Asymmetric Preservation Principle and derive a simple design rule: supervision should follow what survives, rather than reconstruct what is erased. Geometry should guide perception; real demonstrations should ground control. We propose GRA ( G eometry-guided R epresentation A lignment). It routes geometric supervision to the VLA vision backbone via an auxiliary 2D head. Future 2D end-effector waypoints are computed from the source human video through pose estimation, retargeting, physics simulation, and calibrated camera projection, then used to supervise the backbone on generated observations. The action head trains exclusively on real demonstrations. During fine-tuning, the waypoint objective persists as a spatial representation anchor , preventing the backbone from drifting toward action-predictive features at the cost of spatial structure. On real-robot Franka tasks, GRA outperforms pseudo-action baselines under matched data budgets and narrows the gap to policies trained on real robot data alone with substantially more demonstrations. Our contributions are summarized as follows: • Asymmetric Preservation. We formulate the Asymmetric Preservation Principle : generated videos preserve spatial structure more reliably than control dynamics, and should be treated as geometric witnesses rather than control traces. • Supervision Routing. We propose GRA , which routes human-derived geometry to the VLA backbone as future 2D waypoint supervision, independent of synthesized pixels, while learning actions only from real demonstrations. • Representational Anchoring. We demonstrate that action fine-tuning can erode geometry-aware representations; a persistent spatial representation anchor mitigates this drift and is essential to GRA’s effectiveness. 2 Related Work VLA Models and Adaptation VLA models [ 18 , 4 , 12 , 46 , 37 , 31 , 2 , 45 , 44 , 42 ] couple large-scale vision-language pretraining with action prediction and benefit from cross-embodiment data [ 29 , 16 , 40 ] . Early designs cast actions as discrete tokens within an autoregressive multimodal model [ 18 , 46 ] , while more recent variants adopt continuous regression or diffusion-based heads for higher-frequency control [ 17 , 4 , 12 ] . Adapting these models to specific hardware still relies on real teleoperation, which is costly to collect at scale. Recent work therefore investigates parameter-efficient fine-tuning [ 17 , 10 ] and principled data allocation [ 7 ] . We address an orthogonal axis: how to consume synthetic robot videos so that real-data demand is reduced rather than diluted. Synthetic Data for VLA Two sources of synthetic robot data dominate. Simulation provides paired state-action data via domain randomization [ 38 ] or via demo-conditioned generation [ 27 ] , but requires task-specific assets. Human-to-robot video generation [ 41 , 36 , 21 , 9 ] instead synthesizes robot footage from human demonstrations at scale, with no action labels. The dominant paradigm consumes the generated frames by first recovering pseudo-actions, via inverse dynamics [ 13 ] , latent action tokens [ 8 , 43 ] , or geometric retargeting [ 22 , 3 , 24 , 1 ] , and then training the action head on the pseudo-labelled frames. A smaller line instead extracts non-action supervision [ 20 , 23 , 11 , 26 , 32 ] : 2D waypoint regression from edited human videos, sparse path tokens, future-frame prediction, and hand-centric video pretraining. GRA belongs to this second line and applies it to fully synthesized