SEM: Enhancing Spatial Understanding for Robust Robot Manipulation

Xuewu Lin, Tianwei Lin, Yun Du, Yiwei Jin, Jitao Li, Hongyu Xie, Lichao Huang, Zhizhong Su* · 2025

Read SEM: Enhancing Spatial Understanding for Robust Robot Manipulation by Xuewu Lin, Tianwei Lin, Yun Du, Yiwei Jin, Jitao Li, Hongyu Xie, Lichao Huang, Zhizhong Su* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16196.txt.

Introduction

Xuewu Lin and six co-authors present a robot control model in two thousand twenty-five with stronger spatial understanding. Manipulation policies must reason about three-dimensional shape, object relations, and the robot's own body. Point cloud models lack meaning-level grasp of scenes, while image encoders reason poorly about space.

Their model, SEM, is a diffusion-based policy with two added parts. A spatial enhancer enriches image features with three-dimensional geometric context. A robot state encoder models how the robot's joints depend on each other, as a graph. Together they yield robust, generalizable manipulation across varied tasks. The authors report that SEM beats existing baseline methods. The design treats space and embodiment as first-class inputs.

Catalog record

CREATORXuewu Lin, Tianwei Lin, Yun Du, Yiwei Jin, Jitao Li, Hongyu Xie, Lichao Huang, Zhizhong Su*
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16196

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SEM: Enhancing Spatial Understanding for Robust Robot Manipulation Xuewu Lin, Tianwei Lin, Yun Du, Yiwei Jin, Jitao Li, Hongyu Xie, Lichao Huang, Zhizhong Su* Abstract A key challenge in robot manipulation lies in developing policy models with consistent spatial understanding—the ability to reason about 3D geometry, object relations, and robot state. Existing mainstream models take 2D images as input, without performing explicit 3D modeling, and thus lack spatial understanding capabilities as well as 3D and embodiment generalization. To address this, we propose SEM (Spatial Enhanced Manipulation), a diffusion-based policy framework that constructs a unified spatial representation by projecting multi-view image features and joint-centric robot states into a unified 3D space. This spatially aligned representation provides a consistent feature space for the diffusion policy to condition on during action generation. Extensive experiments demonstrate that SEM significantly improves spatial understanding, leading to robust and generalizable manipulation across diverse tasks that outperform existing baselines. I INTRODUCTION Robust robot manipulation in diverse real-world environments hinges on a core capability: spatial understanding—the ability to perceive 3D geometry, object relations, and the robot’s own embodiment states in a consistent way. While recent diffusion-based (or flow-based) policy models have demonstrated strong generative capacity for action modeling, their effectiveness remains limited by perceptual bottlenecks: (1) vision backbones are typically trained for 2D image understanding (visual grounding and visual question answering) and lack explicit 3D reasoning ability, (2) visual features and robot state features are processed independently without spatial alignment, and (3) the robot’s embodiment structure is not effectively modeled. To address these challenges, we propose SEM (Spatial Enhanced Manipulation), a policy framework that constructs a unified spatial representation as a consistent representation space for action generation (Fig. 1 ). First, we introduce a 3D spatial enhancer that injects explicit 3D information into 2D visual features. This module encodes camera intrinsic and extrinsic parameters into the visual representation via 3D absolute position embeddings. Second, when encoding the robot state, we augment the joint angles with the 6D poses, obtained via forward kinematics, and embed these poses as part of the robot state features. Both the 3D absolute position of visual features and the joint poses are expressed in a unified 3D coordinate frame, such as the robot base frame. Finally, to better capture the embodiment structure and support robots with arbitrary numbers of joints, we design a joint-centric robot state encoder and action decoder, which enable feature propagation among joints through graph attention. Robot embodiments and sensor configurations often differ significantly across datasets. When training policies on such heterogeneous data, severe ambiguities can arise: the same joint index may correspond to distinct physical motions on different robots, and varying camera setups can yield observations from inconsistent viewpoints. These ambiguities are a major obstacle to embodiment and sensor generalization in existing models. These ambiguities are the key factor limiting the embodiment and sensor generalization capabilities of existing models. SEM mitigates such inconsistencies by explicitly modeling both the camera and the robot embodiment, thereby achieving 3D and embodiment generalization and obtaining improved performance. Fig. 1: Overview of SEM. SEM constructs a unified spatial representation by projecting multi-view image features and joint-level robot states into the robot base frame. We extensively evaluate SEM in both simulation and real-robot settings. In simulation, we adopt RoboTwin 2.0 benchmark [ 1 ] and evaluate on 16 representative tasks. Despite having only around 40M trainable parameters, SEM significantly outperforms existing approaches. We further conduct targeted experiments on 3D spatial generalization and embodiment generalization, confirming the benefits of our unified spatial representation. In real-world evaluations, SEM consistently outperforms RDT [ 2 ] across all seven tasks. Our contributions are summarized as follows: • Novel Framework. We propose SEM, a new VLA framework, from the perspective of enhancing spatial understanding. • Superior Performance. We evaluate SEM thoroughly in both simulation and real-world robot settings, demonstrating that SEM outperforms existing baselines. • Comprehensive Analysis. We verify the effectiveness of each module in SEM, as well as its improvement in 3D spatial and embodiment generalization. II Related Works End-to-end robot manipulation methods have progressed rapidly, with improvements being made from various perspectives. Early works [ 3 , 4 , 5 , 6 ] use imitation learning to acquire manipulation skills from human demonstrations. Subsequent studies refine action modeling to improve stability and performance: IBC [ 7 ] introduces implicit behavioral cloning, BeT [ 8 ] models multi-modal actions with a Gaussian mixture, and Diffusion Policy (DP) [ 9 ] formulates trajectory generation as a conditional diffusion process, enabling robust learning in high-dimensional action spaces. Moreover, many works have contributed to the improvement of algorithmic pipelines and network architectures. PerAct [ 10 ] encodes voxelized 3D features with a Perceiver transformer to predict end-effector (EE) keyframes. Act3D [ 11 ] introduces coarse-to-fine 3D point sampling for precise action prediction. 3D Diffuser Actor [ 12 ] combines Act3D with diffusion modeling for further gains. RVT [ 13 ] re-renders observations into three orthogonal planes to produce heatmaps of EE keyframes, while RVT-2 [ 14 ] adds a fine branch to boost accuracy. KStarDiffuser [ 15 ] uses graph convolution over joint trajectories with an auxiliary joint position loss for bimanual tasks. However, all these approaches predict EE poses, which limits their ability to handle complex long-horizon or contact-rich tasks such as cloth manipulation. DP3 [ 16 ] addresses this by directly predicting joint positions via diffusion, enabling finer control. Fig. 2: The overall architecture of SEM. SEM takes multi-view images, depth, robot state, and instructions as inputs, projects them into a shared base-frame 3D space, and generates future joint trajectories via a diffusion-based policy. In this way, SEM learns a spatially consistent representation that improves performance and generalization. Recently, breakthroughs in VLMs [ 17 , 18 , 19 , 20 ] have spurred growing research interest in generalist action models. These models [ 21 , 22 , 2 , 23 , 24 , 25 , 26 ] use VLMs as the base model and are trained on diverse embodied intelligence datasets with the aim to achieve high levels of scenario, embodiment, and task generalization. Methods such as SpatialVLA [ 27 ] enhance VLMs with 3D position embeddings for improved spatial reasoning. Despite these advances, VLM-based policies still face challenges in real-world performance and spatial generalization. Our work focuses more on network design for improved 3D spatial understanding rather than concentrating on VLM-based generalist models. The most closely related approaches are KStarDiffuser [ 15 ] and SpatialVLA [ 27 ] . Unlike SEM, both of them output EE poses rather than joint trajectories, and SpatialVLA does not model camera extrinsic or unify visual and action representations in a common 3D space. III Problem Formulation The inputs for the manipulation problem primarily consist of sensor data and current joint positions S , where the sensor data generally encompass multi-view images I and depth maps D . To simplify the problem, this paper utilizes only the sensor data from the current frame. When handling multitasks, instruction texts \mathrm{T} are also required. Moreover, we consider modeling both the camera and the embodiment structure, thus requiring inputs of camera parameters C and embodiment structure parameters E . Thus, the inputs can be summarized into the following parts:

Read the complete text