ST-VLA: Enabling 4D-Aware Spatiotemporal Understanding for General Robot Manipulation

You Wu, Zixuan Chen, Cunxu Ou, Wenxuan Wang, Wenbo Huang, Lin Cao, Yangtao Chen, Weichao Qiu, Xingyue Quan, Jieqi Shi, Jing Huo, Yang Gao · 2026

Read ST-VLA: Enabling 4D-Aware Spatiotemporal Understanding for General Robot Manipulation by You Wu, Zixuan Chen, Cunxu Ou, Wenxuan Wang, Wenbo Huang, Lin Cao, Yangtao Chen, Weichao Qiu, Xingyue Quan, Jieqi Shi, Jing Huo, Yang Gao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13788.txt.

Introduction

You Wu and co-authors propose ST-VLA, a hierarchical VLA framework with 4D-aware spatiotemporal understanding for robot manipulation. Past VLA frameworks use 2D representations, which lack depth awareness and temporal consistency in complex 3D scenes. ST-VLA turns 2D guidance into 3D trajectories and smooth spatial masks that carry 4D context. That gives a stable interface between semantic reasoning and continuous control. The team also built ST-Human, a dataset of 14 tasks and 300k episodes with 2D, 3D, and 4D supervision.

Catalog record

CREATORYou Wu, Zixuan Chen, Cunxu Ou, Wenxuan Wang, Wenbo Huang, Lin Cao, Yangtao Chen, Weichao Qiu, Xingyue Quan, Jieqi Shi, Jing Huo, Yang Gao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62320
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13788

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ST-VLA: Enabling 4D-Aware Spatiotemporal Understanding for General Robot Manipulation You Wu, Zixuan Chen, Cunxu Ou, Wenxuan Wang, Wenbo Huang, Lin Cao, Yangtao Chen, Weichao Qiu, Xingyue Quan, Jieqi Shi, Jing Huo, Yang Gao Abstract Robotic manipulation in open-world environments requires reasoning across semantics, geometry, and long-horizon action dynamics. Existing hierarchical Vision-Language-Action (VLA) frameworks typically use 2D representations to connect high-level reasoning with low-level control, but lack depth awareness and temporal consistency, limiting robustness in complex 3D scenes. We propose ST-VLA , a hierarchical VLA framework using a unified 3D-4D representation to bridge perception and action. ST-VLA converts 2D guidance into 3D trajectories and generates smooth spatial masks that capture 4D spatio-temporal context, providing a stable interface between semantic reasoning and continuous control. To enable effective learning of such representations, we introduce ST-Human , a large-scale human manipulation dataset with 14 tasks and 300k episodes, annotated with 2D, 3D, and 4D supervision via a semi-automated pipeline. Using ST-Human, we train ST-VLM , a spatio-temporal vision-language model that generates spatially grounded and temporally coherent 3D representations to guide policy execution. The smooth spatial masks focus on task-relevant geometry and stabilize latent representations, enabling online replanning and long-horizon reasoning. Experiments on RLBench and real-world manipulation tasks show that ST-VLA significantly outperforms state-of-the-art baselines, improving zero-shot success rates by 44.6% and 30.3%. These results demonstrate that offloading spatio-temporal reasoning to VLMs with unified 3D-4D representations substantially improves robustness and generalization for open-world robotic manipulation. Project website: https://oucx117.github.io/ST-VLA/ . 1 1 footnotetext: Equal contribution. Figure 1 : ST-VLM bridges the semantic-physical gap via unified 3D-4D spatio-temporal representations. (Left) Existing 2D-based VLMs face geometric ambiguity and temporal inconsistency due to the semantic-physical mismatch. (Right) Our ST-VLA utilizes unified 3D-4D representations with explicit trajectories and smooth spatial masks, ensuring robust long-horizon manipulation. 1 Introduction Robust robot manipulation in open-world environments requires tightly integrating generalizable semantic understanding, accurate spatial perception, and reliable execution. Beyond identifying interaction targets, robots must reason about how and where to act in complex three-dimensional (3D) environments. Hierarchical Vision-Language-Action (VLA) architectures have therefore emerged as a dominant paradigm, where high-level Vision-Language Models (VLMs) perform task understanding and semantic reasoning, and low-level policies execute continuous control, improving generalization across manipulation tasks ( 4 ; 60 ; 12 ; 27 ; 1 ; 42 ; 9 ) . Despite this progress, existing VLMs are primarily trained in large-scale two-dimensional (2D) images and visual question answering data, limiting their ability to model 3D geometry and long-horizon manipulation dynamics in physical environments ( 40 ; 31 ) . As a result, high-level semantic predictions often fail to align with the continuous 3D execution space required for fine-grained robotic control, leading to deviations from action, instability, and task failure ( 1 ; 20 ) . To alleviate this issue, previous work introduces intermediate representations such as 2D waypoints, bounding boxes, or segmentation masks to connect semantic reasoning with low-level control ( 30 ; 15 ; 43 ; 23 ; 2 ; 53 ; 45 ) . However, these interfaces remain fundamentally constrained by the 2D paradigm. Planar representations cannot capture depth and geometric constraints, and frame-wise signals ignore temporal continuity ( 24 ; 20 ; 10 ; 32 ; 14 ) . Consequently, they lack explicit modeling of 4D (3D + time) spatio-temporal consistency, often inducing action jitter and reduced robustness in dynamic or long-horizon tasks ( 59 ) . We argue that these limitations stem from a fundamental representation mismatch between high-level semantic spaces and low-level physical spaces ( 12 ; 20 ; 25 ; 39 ; 22 ) . VLM representations are rooted in static, projective visual domains, whereas robot control operates in continuous 3D physical environments. As shown in Figure 1 (a),the 2D intermediates form a lossy bridge that discards critical geometric and temporal cues. This forces low-level policies to infer missing dimensions from ambiguous signals, leading to spatial misalignment and execution inaccuracy under novel viewpoints or disturbances ( 20 ; 24 ) , as well as temporal incoherence that hinders long-horizon stability ( 8 ; 7 ) . An effective intermediate representation should instead jointly capture 3D geometry, temporal continuity, and cross-modal consistency. Motivated by this insight, we propose ST-VLA , a hierarchical VLA framework that bridges the perception-action gap via unified 3D-4D representations. As shown in Figure 1 (b), instead of rigid 2D cropping, ST-VLA lifts conventional 2D guidance to explicit 3D trajectories and constructs cross-modally aligned smooth spatial masks to inpaint task-irrelevant distractors. This smooth alignment ensures focus on task-relevant geometry, which preserves latent continuity, suppresses model hallucinations, and improves action stability. Moreover, by integrating 4D spatio-temporal context, ST-VLA enables online replanning and robust long-horizon execution. To enable high-level VLMs to produce 3D-4D representations, we introduce ST-Human , a large-scale dataset with 4.3M samples, built from 300K continuous human manipulation trajectories paired with high-fidelity 3D observations. Unlike prior 2D-centric or static datasets, ST-Human captures depth, object geometry, and temporal action evolution, providing unified spatio-temporal supervision. Building on ST-Human, we propose a unified 2D-3D-4D task learning framework that extends VLM semantic reasoning across 3D spatial and 4D temporal domains. By jointly fine-tuning a 4B Qwen3-VL model ( 51 ) on ST-Human alongside specialized public datasets—including RoboPoint ( 53 ) , FSD ( 54 ) , and SAT ( 41 ) —we develop ST-VLM. This unified training approach integrates 2D trajectory grounding, depth-aware 3D perception, and long-horizon 4D reasoning, enabling the model to acquire comprehensive spatio-temporal understanding within a single representative space. With explicit geometric and temporal constraints, ST-VLM learns stable, transferable 3D–4D reasoning, enabling reliable zero-shot guidance for open-world manipulation. We evaluate ST-VLM and ST-VLA on multiple benchmark datasets as well as simulated and real-world robotic platforms. On the RoboRefit ( 33 ) , CVBench ( 48 ) , and SAT ( 41 ) datasets, ST-VLM achieves up to 33.19% improvement over existing methods. On 3 representative RLBench tasks, ST-VLA improves zero-shot success rates by 44.6%. Real-world long-horizon experiments further demonstrate its execution stability and cross-scenario generalization. Our contributions are summarized as follows: (1) We propose ST-VLA , a hierarchical Vision-Language-Action (VLA) framework built upon unified 3D-4D (3D + time) intermediate representations. ST-VLA replaces conventional, error-prone 2D priors with explicit 3D trajectories and latent 4D spatio-temporal context, ensuring action stability and suppressing hallucinations. (2) We introduce ST-Human , a high-fidelity 3D-4D human manipulation dataset,and propose a unified 2D-3D-4D task learning and evaluation framework. Based on this framework, we train a 4B-parameter vision-language model, ST-VLM , which acquires transferable 3D-4D reasoning capabilities for zero-shot open-world manipulation. (3) We evaluate ST-VLM and ST-VLA on multiple benchmarks and simulated and real-world robotic platforms. Results show clear gains in zero-shot success rate, long-horizon stability, and cross-scenario generalization over existing methods. Figure 2 : Overview of the ST-Human Dataset Construction and Unified 2D-3D-4D Task Generation. 2 Related Work 2.1 Point and Trajectory-based Spatial Guidance An important line of research in manipulation focuses on the generalization of robotic policies by providing explicit spatial guidance. Early works utilize 2D waypoints or bounding boxes ( 30 ; 19 ) to anchor the policy’s attention to specific image-plane coordinates. Beyond image-plane cues, another line explores 3D value-based spatial representations as guidance for manipulation ( 6 ; 20 ) . More recently, trajectory-based task

Read the complete text