Learning Geometrically-Grounded 3D Visual Representations for View-Generalizable Robotic Manipulation
Di Zhang, Weicheng Duan, Dasen Gu, Hongye Lu, Hai Zhang, Hang Yu, Junqiao Zhao, Guang Chen, [1.5mm] Tongji University · 2026
Read Learning Geometrically-Grounded 3D Visual Representations for View-Generalizable Robotic Manipulation by Di Zhang, Weicheng Duan, Dasen Gu, Hongye Lu, Hai Zhang, Hang Yu, Junqiao Zhao, Guang Chen, [1.5mm] Tongji University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22988.txt.
Introduction
Di Zhang and a team at Tongji University teach robots to see scenes in three dimensions. Real manipulation needs a steady grasp of space, even when cameras move to new viewpoints. Policies trained from one angle often fail when the view changes.
Their method learns visual representations grounded in geometry, so scenes read consistently across views. A robot that understands shape, not just pixels, can pick and place objects more reliably. View-proof vision is a step toward helpers that work in messy, real rooms. Robust spatial understanding is the stated requirement, because a manipulation policy must know where things are, not just what they look like.
Catalog record
| CREATOR | Di Zhang, Weicheng Duan, Dasen Gu, Hongye Lu, Hai Zhang, Hang Yu, Junqiao Zhao, Guang Chen, [1.5mm] Tongji University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22988
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Geometrically-Grounded 3D Visual Representations for View-Generalizable Robotic Manipulation Di Zhang, Weicheng Duan, Dasen Gu, Hongye Lu, Hai Zhang, Hang Yu, Junqiao Zhao, Guang Chen, [1.5mm] Tongji University Abstract ∗ Equal contribution. Contact {2331922, 2252109}@tongji.edu.cn † Corresponding author. Contact [email protected] Project Website: gem3d-project.github.io/gem3d.github.io/ Real-world robotic manipulation demands visuomotor policies capable of robust spatial scene understanding and strong generalization across diverse camera viewpoints. While recent advances in 3D-aware visual representations have shown promise, they still suffer from several key limitations: (i) reliance on multi-view observations during inference, which is impractical in single-view restricted scenarios; (ii) incomplete scene modeling that fails to capture holistic and fine-grained geometric structures essential for precise manipulation; and (iii) lack of effective policy training strategies to retain and exploit the acquired 3D knowledge. To address these challenges, we present GEM3D ( Ge o m etrically-Grounded 3D Manipulation), a unified representation-policy learning framework for view-generalizable robotic manipulation. GEM3D introduces a single-view 3D pretraining paradigm that leverages point cloud reconstruction and feed-forward gaussian splatting under multi-view supervision to learn holistic geometric representations. During policy learning, GEM3D performs multi-step distillation to preserve the pretrained geometric understanding and effectively transfer it to manipulation skills. We conduct experiments on 12 RLBench tasks, where our approach outperforms the previous state-of-the-art (SOTA) method by 12.7% in average success rate. Further evaluation on six representative tasks demonstrates the strong zero-shot view generalization of our approach, with the success rate drops by only 22.0% and 29.7% under moderate and large viewpoint shifts, respectively, whereas the SOTA method suffers larger decreases of 41.6% and 51.5% . (a) GEM3D learns fine-grained 3D representations that enable more accurate novel-view rendering and point cloud reconstruction. (b) GEM3D exhibits strong robustness to viewpoint shifts, maintaining stable visuomotor performance during inference. Figure 1 : We present GEM3D , a unified rerepresentation-policy learning framework for view-generalizable robotic manipulation. 1 Introduction Learning end-to-end visuomotor policies enables robots to perceive their surroundings and act upon visual understanding 3 ; 18 ; 4 ; 12 ; 45 ; 9 ; 11 . However, real-world environments are often spatially complex and partially occluded, with varying camera viewpoints. Therefore, learning spatially grounded and view-invariant visual representations is essential for developing robust visuomotor policies. Most existing approaches 41 ; 19 ; 10 employ 2D visual encoders to compress images into latent vectors that capture task-relevant states. However, such 2D features lack explicit 3D structural awareness, limiting their effectiveness in spatially intricate manipulation tasks. Recent studies have attempted to address this limitation by leveraging multi-view inputs to learn 3D-aware visual representations 7 ; 28 ; 32 . While promising, such methods are often impractical for real-world deployment due to hardware constraints and inference overhead. More recent efforts 6 ; 37 ; 23 ; 43 ; 44 introduce single-view 3D perception by incorporating auxiliary reconstruction objectives such as Neural Radiance Fields (NeRFs) 24 or Gaussian Splatting 17 . Although these methods enhance 3D understanding, their scene modeling remains coarse, often failing to recover fine-grained geometric structures that are crucial for precise manipulation (as shown in Fig. 1 ). Moreover, the resulting visual encoders tend to overfit to specific training viewpoints, leading to degraded generalization in unseen configurations during inference. To learn more expressive 3D manipulation representations with high-fidelity scene embeddings and strong generalization across varying camera viewpoints, we propose GEM3D ( Ge o m etrically-Grounded 3D Manipulation), a unified representation-policy learning framework that leverages geometrically-grounded 3D representation pretraining on multi-view data and distills the acquired geometric knowledge into a single-view visual policy for downstream robotic manipulation tasks. GEM3D adopts a three-stage pretraining paradigm to learn expressive 3D scene embeddings: (i) From single-view RGB-D observations, the model encodes the input into a voxel-based Dense Volumetric Feature . (ii) This volumetric representation is then used to predict skeleton seed points, which are progressively refined into complete point structures via a coarse-to-fine Snowflake process 38 , enabling holistic geometric understanding. (iii) Guided by the refined points, the Dense Volumetric Feature is resampled to predict Gaussian primitives for novel-view rendering under multi-view supervision, capturing fine-grained appearance and texture details. Through this pretraining paradigm, GEM3D learns to infer the complete geometry and appearance of a scene from single-view inputs, thereby achieving robust scene understanding and strong view-generalization. Beyond the generalizable Dense Volumetric Feature , developing a robust policy that fully retains the scene understanding capability acquired during pretraining is equally crucial. To this end, we propose a multi-step distillation-based visuomotor policy that can integrate with any pretrained representations, featuring an implicit dynamics-aware design that enhances perception-action consistency. Rather than directly fine-tuning the pretrained encoder, GEM3D softly guides the policy’s visual encoder training by aligning its latent features with the pretrained representations, effectively transferring the acquired geometric understanding to manipulation skills. To evaluate the effectiveness of GEM3D , we conduct comprehensive experiments on 12 RLBench manipulation tasks 16 . GEM3D achieves a 12.7% improvement in average success rate over the previous state-of-the-art (SOTA) method ManiGaussian 23 . It also demonstrates strong zero-shot view generalization, with success rates dropping by only 22.03% and 29.67% under moderate and large viewpoint shifts, compared to the SOTA method’s 41.62% and 51.52% declines. In addition, integrating our distillation-based policy with ManiGaussan-Pretrained representation backbones yields consistent performance gains, indicating its potential for diversified extension to a broader range of backbone architectures. Ablation studies further verify the necessity of each component in our framework. Figure 2 : Overview of GEM3D . GEM3D comprises two key components: (1) GEM3D Pretraining , which learns holistic 3D representations through auxiliary scene reconstruction tasks; and (2) GEM3D Policy , which distills the pretrained 3D visual representations into a visuomotor policy for view-generalizable manipulation. 2 Related Work 2.1 3D Representation Learning Recent advances in visuomotor policy learning have explored diverse representation forms, ranging from 2D image features 3 ; 30 to 3D structures such as voxel grids 33 and point clouds 2 ; 8 ; 44 . Among these, novel view synthesis (NVS) has recently gained attention as an auxiliary representation learning objective, as it enables models to infer unseen viewpoints from limited observations—reflecting a holistic understanding of 3D geometry and spatial consistency 24 ; 17 ; 27 ; 22 ; 31 . However, achieving accurate and generalizable reconstruction remains challenging for existing NVS paradigms. Classical methods such as NeRFs 24 and Gaussian Splatting 17 often struggle to generalize beyond the training distribution, leading to degraded rendering quality in novel scenes or viewpoints. More recent feed-forward Gaussian Splatting approaches—ImageSplatter 34 , VoxSplat 29 , and PixelSplat 1 —improve generalization but remain unstable in dynamic environments, exhibiting noticeable degradation in reconstructed geometry and texture (see Fig. 1 ). These limitations highlight the need for a NVS framework that achieves both high-fidelity reconstruction and robust generalization across diverse robotic scenes. 2.2 Visuomotor Policy Learning To effectively leverage the benefits of auxiliary representation learning tasks, recent visuomotor policy research can be categorized into two major training paradigms: (i) Pretrain-Finetune . These methods leverage visual encoders that are pretrained on large-scale 2D or 3D perception datasets and subsequently finetuned for downstream control tasks 39 ; 36 ; 40 ; 42 ; 28 . However, domain discrepancies between pretraining and downstream control tasks can lead to suboptimal transfer, and even degrade policy learning, as