Point Bridge : 3D Representations for Cross Domain Policy Learning

Siddhant Haldar Lars Johannsmeier Lerrel Pinto Abhishek Gupta Dieter Fox Yashraj Narang Ajay Mandlekar NVIDIA New York University University of Washington https://pointbridge3d.github.io/ · 2026

Read Point Bridge : 3D Representations for Cross Domain Policy Learning by Siddhant Haldar Lars Johannsmeier Lerrel Pinto Abhishek Gupta Dieter Fox Yashraj Narang Ajay Mandlekar NVIDIA New York University University of Washington https://pointbridge3d.github.io/ (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16212.txt.

Introduction

Siddhant Haldar and coauthors use point forms to move robot skills across fields. Their 2026 preprint deals with a lack of real robot task data. Fake and game data are cheap to make at scale. But a look gap parts fake views from real views. Skills learned in fake worlds then fail in real rooms.

Point forms strip views to bare 3D point sets. Bare points look more like in both worlds than full pics do. Skills taught on points may cross the gap with less loss. The team tests point skills on real tasks. The work, called Point Bridge, seeks cheap paths to real skill.

Catalog record

CREATORSiddhant Haldar Lars Johannsmeier Lerrel Pinto Abhishek Gupta Dieter Fox Yashraj Narang Ajay Mandlekar NVIDIA New York University University of Washington https://pointbridge3d.github.io/
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16212

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Point Bridge : 3D Representations for Cross Domain Policy Learning Siddhant Haldar Lars Johannsmeier Lerrel Pinto Abhishek Gupta Dieter Fox Yashraj Narang Ajay Mandlekar NVIDIA New York University University of Washington https://pointbridge3d.github.io/ Abstract Robot foundation models are beginning to deliver on the promise of generalist robotic agents, yet progress remains constrained by the scarcity of large-scale real-world manipulation datasets. Simulation and synthetic data generation offer a scalable alternative, but their usefulness is limited by the visual domain gap between simulation and reality. In this work, we present Point Bridge , a framework that leverages unified, domain-agnostic point-based representations to unlock synthetic datasets for zero-shot sim-to-real policy transfer—without explicit visual or object-level alignment. Point Bridge combines automated point-based representation extraction via Vision-Language Models (VLMs), transformer-based policy learning, and efficient inference-time pipelines to train capable real-world manipulation agents using only synthetic data. With additional co-training on small sets of real demonstrations, Point Bridge further improves performance, substantially outperforming prior vision-based sim-and-real co-training methods. It achieves up to 44% gains in zero-shot sim-to-real transfer and up to 66% with limited real data across both single-task and multitask settings. Videos of the robot are best viewed at: https://pointbridge3d.github.io/ . 1 Introduction Deep learning has recently undergone a paradigm shift, moving from narrow task-specific models to generalist systems capable of complex reasoning ( 1 ; 70 ; 73 ) , generating photorealistic images ( 10 ) and videos ( 46 ) , and even writing code ( 42 ) . This progress has been fueled by internet-scale training data paired with scalable architectures. Lately, robot foundation models are starting to realize some of the promise of large-scale data and the training paradigm from these domains. However, unlike vision and language, which can directly exploit internet-scale datasets, robotics is inherently interactive: models must learn from datasets that contain physical interactions with the real world. This makes collecting large-scale robotic data time-consuming, prohibitively expensive, and fundamentally difficult to scale, creating a central bottleneck for building generalist robotic intelligence. The prevailing paradigm for robot policy learning relies on large-scale teleoperated datasets, followed by training neural policies on them. While effective, this approach often requires months or years of data collection and still produces datasets far smaller than those in vision and language ( 24 ) . Simulation is a promising alternative to address this need for data, especially due to recent progress. Simulation environments are becoming easier to design, with the availability of high-fidelity physics simulators ( 71 ; 55 ) and the emergence of generative AI tools that automate asset and scene generation ( 77 ; 58 ) . Recently developed synthetic data generation tools can generate large-scale, high-quality robot manipulation demonstration datasets in such simulation environments with minimal human effort ( 16 ; 50 ; 33 ; 23 ) . Furthermore, recent work has shown that such synthetic simulation datasets can easily train high-performing real-world manipulation agents by co-training on these datasets and small numbers of real-world demonstrations ( 49 ; 78 ; 9 ) , suggesting that synthetic simulation data could potentially reduce the dependence on large real-world datasets. However, these methods can still require careful sim and real alignment, and still rely on the presence of real-world data, owing to the mismatched representation of data between the domains. Human videos offer another scalable and complementary source of supervision, but again face challenges from the embodiment gap between human and robot morphologies as well as the representation mismatch between the domains. A recent line of work proposes task-relevant keypoint representations ( 27 ; 91 ; 45 ) as a potential solution to this domain representation gap. By abstracting both the robot and scene into sets of keypoints, these methods enable policies that are agnostic to raw visual appearance and generalize across objects and environment conditions. However, existing approaches often rely on human annotations ( 27 ; 45 ) , focus on bridging embodiment but not visual differences ( 39 ; 38 ) , and are often restricted to single-task settings. We argue that such representations only scratch the surface of what is possible. In this work, we introduce Point Bridge , a framework that uses unified domain-agnostic point-based representations to unlock the potential of synthetic simulation datasets and enable zero-shot sim-to-real policy transfer. Point Bridge trains real-world manipulation agents starting with just a handful of teleoperated demonstrations in simulation by using synthetic data generation tools. It then leverages advances in vision-language models (VLMs) to build unified scene representations that facilitate cross-domain policy transfer. Our core insight is that unifying representations across simulation and real-robot teleoperation unlocks scalable sim-to-real transfer without requiring explicit visual or object-level alignment. Such a representation further supports scaling to multi-task policies through transformer-based architectures, providing a framework that scales with data availability. Point Bridge operates in three stages. First, scenes are filtered into point cloud–based representations aligned to a common reference frame. In simulation, this is obtained directly from object meshes, while in real experiments, we use our automated VLM-guided pipeline for keypoint extraction on task relevant objects. Second, a transformer-based policy architecture is trained on these unified point clouds for policy learning. Finally, during deployment, we employ a lightweight pipeline for scene extraction designed to minimize the sim-to-real gap, leveraging VLM filtering and supporting multiple 3D sensing strategies to balance performance and throughput. We demonstrate the effectiveness of Point Bridge on six real-world tasks, using data collected through simulation and real robot teleoperation. Our main findings are as follows: 1. We develop Point Bridge , a framework that uses unified domain-agnostic point-based representations to harness synthetic simulation data and enable zero-shot sim-to-real policy transfer. 2. Point Bridge contains novel components including (1) a VLM-based point extraction pipeline that bridges the visual sim-to-real gap with minimal human effort, and (2) multiple inference-time pipelines to adapt to different user needs with respect to performance and throughout. 3. Point Bridge improves by 39% and 44% on single-task and multitask zero-shot sim-to-real transfer. When co-trained with a small amount of real data, Point Bridge improves over prior works by 61% and 66% in single-task and multitask settings, respectively. (Section 5.2 , 5.3 ). 4. We present a systematic analysis of key design choices in Point Bridge (Section 5.4 ). All of our datasets, training, and evaluation code will be made publicly available. Videos of our trained policies are best viewed at: https://pointbridge3d.github.io/ . 2 Related Work 2.1 Structured Representations Structured representations of scene elements enable more efficient and semantically meaningful learning. Common techniques include segmentation into bounding boxes ( 18 ; 90 ) and object pose estimation ( 74 ; 75 ) . Bounding boxes show promise but suffer from overfitting to specific instances, while pose estimation is less prone to this but requires separate models per object. Point clouds ( 88 ; 4 ) are a popular alternative but their unstructured nature complicates learning spatial relationships. Recently, key points ( 41 ; 34 ; 29 ; 27 ; 22 ; 65 ) have gained traction for policy learning due to their generalizability and support for direct human prior injection ( 7 ; 6 ) , contrasting with approaches that first learn representations from human videos followed by robot teleoperation data ( 57 ;

Read the complete text