One View, Many Worlds: Single-Image to 3D Object Meets Generative Domain Randomization for One-Shot 6D Pose Estimation
Zheng Geng, Nan Wang, Shaocong Xu, Chongjie Ye, Bohan Li, Zhaoxi Chen, Sida Peng, Hao Zhao, [email protected] [email protected] · 2025
Read One View, Many Worlds: Single-Image to 3D Object Meets Generative Domain Randomization for One-Shot 6D Pose Estimation by Zheng Geng, Nan Wang, Shaocong Xu, Chongjie Ye, Bohan Li, Zhaoxi Chen, Sida Peng, Hao Zhao, [email protected] [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07978.txt.
Introduction
Estimating an object's exact position and orientation from one reference photo is the challenge met by Zheng Geng and 7 co-authors in a 2025 computer vision paper. Robots meet endless unfamiliar objects, and usually no 3D model exists. Single-image reconstructions also lack true scale, and generated models differ from real camera views.
Their pipeline, OnePoseViaGen, first aligns scale and pose in a coarse-to-fine way, combining multi-view feature matching with render-and-compare refinement. A text-guided generative step randomises the training domain so the estimator survives the gap between synthetic and real images. One-shot pose estimation would let warehouse and home robots handle new products and tools without lengthy enrolment for every item.
Catalog record
| CREATOR | Zheng Geng, Nan Wang, Shaocong Xu, Chongjie Ye, Bohan Li, Zhaoxi Chen, Sida Peng, Hao Zhao, [email protected] [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07978
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
One View, Many Worlds: Single-Image to 3D Object Meets Generative Domain Randomization for One-Shot 6D Pose Estimation Zheng Geng, Nan Wang, Shaocong Xu, Chongjie Ye, Bohan Li, Zhaoxi Chen, Sida Peng, Hao Zhao, [email protected] [email protected] Abstract Estimating the 6D pose of arbitrary unseen objects from a single reference image is critical for robotics operating in the long-tail of real-world instances. However, this setting is notoriously challenging: 3D models are rarely available, single-view reconstructions lack metric scale, and domain gaps between generated models and real-world images undermine robustness. We propose OnePoseViaGen, a pipeline that tackles these challenges through two key components. First , a coarse-to-fine alignment module jointly refines scale and pose by combining multi-view feature matching with render-and-compare refinement. Second , a text-guided generative domain randomization strategy diversifies textures, enabling effective fine-tuning of pose estimators with synthetic data. Together, these steps allow high-fidelity single-view 3D generation to support reliable one-shot 6D pose estimation. On challenging benchmarks (YCBInEOAT, Toyota-Light, LM-O), OnePoseViaGen achieves state-of-the-art performance far surpassing prior approaches. We further demonstrate robust dexterous grasping with a real robot hand, validating the practicality of our method in real-world manipulation. Project page: https://gzwsama.github.io/OnePoseviaGen.github.io/ Figure 1: OnePoseViaGen. (Left) From a single anchor image, we generate a textured 3D model that lacks real-world scale and pose. Our coarse-to-fine alignment resolves this challenge, and optional domain-randomized fine-tuning further boosts robustness. (Right) While prior methods (Oryon, LoFTR, Gedi) largely fail in this one-shot setting, our approach achieves dramatic gains (ADD 81.3, ADD-S 93.1), enabling reliable 6D pose estimation where baselines break down. Keywords: Object Pose Estimation, Generative Model, Robot Manipulation 1 Introduction Robots operating in the open world must interact with an endless long-tail of novel objects—ranging from an unseen tool on a factory line to an ad-hoc household item in daily life. At the core of this capability lies 6D object pose estimation, the recovery of an object’s precise 3D position t and orientation R . However, this task remains notoriously difficult in realistic settings: pre-scanned CAD models are rarely available, multi-view capture is impractical, and single-view reconstructions suffer from scale ambiguity. Consequently, reliable one-shot 6D pose estimation from a single image has long been deemed nearly impossible, despite its central role in simulation [ 1 , 2 , 3 ] , tracking [ 4 ] , reconstruction [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 ] , and downstream robotic applications such as manipulation [ 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 ] , navigation [ 46 , 47 ] , autonomous control [ 48 , 49 ] and teleoperation [ 50 , 51 ] . While VLA models and generalist policies [ 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 ] excel in broad skills such as folding or bussing, they often fail in precision-critical operations—for example, inserting a charging plug—where sub-centimeter errors lead to task failure. Accurate 6D pose estimation is thus indispensable for reliable robotics. Most state-of-the-art 6D pose estimation methods are learning-based and can be classified into three categories: instance-level [ 62 , 63 , 64 , 65 ] , category-level [ 66 , 67 , 68 , 69 , 70 , 71 ] , and category-agnostic [ 72 , 73 , 74 , 75 ] . While instance-level methods achieve high accuracy, they are restricted to objects with rich texture in the training set, limiting their use in long tail applications. Category-level methods generalize better within known categories but still struggle with unseen object classes. Recent advances in category-agnostic methods focus on pose estimation for unseen objects using render&compare strategies [ 74 , 76 ] , feature matching [ 77 , 73 ] , or both [ 72 ] , achieving high precision. However, these methods depend on 3D models, which are not trivial to acquire in most scenarios. To tackle the lack of object models, recent work in model-free pose estimation explores methods that avoid reliance on explicit textured 3D models. These approaches use reference images and have shown promising results, with some employing partial matching for relative pose estimation [ 78 , 79 , 80 ] or multiple images to reconstruct the object [ 74 , 81 ] . However, many still require multi-view inputs or prior camera-to-object pose knowledge [ 80 , 72 ] , limiting their effectiveness when only a single image is available but robust 6D pose estimation is needed. To address these challenges, we propose OnePoseViaGen, a novel model-free method for 6D object pose estimation. Building on single-image 3D generation techniques [ 82 , 83 , 84 ] , we extend Hi3DGen [ 82 ] with image-conditioned generation to reconstruct textured 3D object models from a single RGB-D image. Since such reconstructions are scale-normalized, we introduce a coarse-to-fine strategy that jointly refines object scale and 6D pose for accurate alignment. By integrating the refined model with advanced pose estimation frameworks [ 74 ] , our method achieves robustness under heavy occlusion, complex lighting, and large viewpoint variations. We evaluate OnePoseViaGen on standard benchmarks (YCBInEOAT, Toyota-Light, LM-O) and a new annotated in-the-wild dataset, and further demonstrate its effectiveness in dexterous grasping tasks. Results show that our method significantly outperforms state-of-the-art approaches for unseen objects. Our contributions are summarized as follows: • Generative pipeline for one-shot 6D pose. We introduce the first pipeline that integrates single-view 3D generation into both training and inference for one-shot 6D pose and scale estimation, proving that generative modeling can directly benefit pose estimation. • Coarse-to-fine metric alignment. We design a coarse-to-fine alignment module that jointly refines pose and metric scale via 2D–3D feature matching and render-and-compare refinement, enabling accurate metric-scale alignment from a single image. • Text-guided generative domain randomization. We propose a text-driven augmentation strategy that generates diverse, structurally consistent 3D variants. Rendering these models under randomized conditions yields a large-scale synthetic dataset, which bridges the domain gap between generated models and real-world objects. • State-of-the-art performance and real-world validation. Our method establishes new state-of-the-art results on three challenging 6D pose benchmarks and a newly collected in-the-wild dataset. We further demonstrate its effectiveness in dexterous robotic grasping, validating robustness and practicality in real-world manipulation. 2 Related Works 2.1 6D Pose Estimation The field of 6D pose estimation has been extensively explored through both model-based and model-free approaches, each addressing the challenge of localizing objects in 3D space with varying strategies [ 85 , 86 , 87 , 88 , 76 , 89 , 73 , 72 , 74 ] . Model-based methods typically rely on CAD models or 3D representations for pose estimation. For instance, MegaPose [ 76 ] employs a render&compare strategy to refine poses of novel objects using synthetic views of CAD models. Similarly, FoundPose [ 73 ] leverages foundation features [ 90 , 91 , 92 ] from self-supervised