6DOPE-GS: Online 6D Object Pose Estimation using Gaussian Splatting
Yufeng Jin, Vignesh Prasad, Snehal Jauhri, Mathias Franzius, Georgia Chalvatzaki · 2024
Read 6DOPE-GS: Online 6D Object Pose Estimation using Gaussian Splatting by Yufeng Jin, Vignesh Prasad, Snehal Jauhri, Mathias Franzius, Georgia Chalvatzaki (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01543.txt.
Introduction
Yufeng Jin and co-authors present a method that tracks an object's full pose live from one depth camera, in a 2024 paper. Robots and augmented reality need to know where things sit and how they tilt. Model-free pose tracking of new objects is costly to compute. Rendering guesses and comparing them each frame eats the budget.
Gaussian Splatting, a fast rendering technique, changes that math. The system optimizes pose and a three-dimensional object model together. Differentiable rendering lets each frame refine both at once. No prior model of the object is required. Live performance suits interactive applications. Robotics teams gain a tracker that learns the object on the fly. The approach may extend to cluttered multi-object scenes.
Catalog record
| CREATOR | Yufeng Jin, Vignesh Prasad, Snehal Jauhri, Mathias Franzius, Georgia Chalvatzaki |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-01543
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
6DOPE-GS: Online 6D Object Pose Estimation using Gaussian Splatting Yufeng Jin, Vignesh Prasad, Snehal Jauhri, Mathias Franzius, Georgia Chalvatzaki Abstract Efficient and accurate object pose estimation is an essential component for modern vision systems in many applications such as Augmented Reality, autonomous driving, and robotics. While research in model-based 6D object pose estimation has delivered promising results, model-free methods are hindered by the high computational load in rendering and inferring consistent poses of arbitrary objects in a live RGB-D video stream. To address this issue, we present 6DOPE-GS, a novel method for online 6D object pose estimation & tracking with a single RGB-D camera by effectively leveraging advances in Gaussian Splatting. Thanks to the fast differentiable rendering capabilities of Gaussian Splatting, 6DOPE-GS can simultaneously optimize for 6D object poses and 3D object reconstruction. To achieve the necessary efficiency and accuracy for live tracking, our method uses incremental 2D Gaussian Splatting with an intelligent dynamic keyframe selection procedure to achieve high spatial object coverage and prevent erroneous pose updates. We also propose an opacity statistic-based pruning mechanism for adaptive Gaussian density control, to ensure training stability and efficiency. We evaluate our method on the HO3D and YCBInEOAT datasets and show that 6DOPE-GS matches the performance of state-of-the-art baselines for model-free simultaneous 6D pose tracking and reconstruction while providing a 5 × \times speedup. We also demonstrate the method’s suitability for live, dynamic object tracking and reconstruction in a real-world setting. 1 Introduction Precise tracking and accurate reconstruction of objects allows capturing essential spatial and structural information, essential for downstream tasks such as robotic manipulation [ 9 , 56 ] , augmented reality [ 57 , 77 ] , automation [ 26 , 14 ] , etc. The majority of 6D object pose estimation and tracking methods, whether for seen or unseen objects, have primarily used model-based techniques. Several approaches [ 41 , 79 , 28 , 30 , 72 ] use CAD models rendered from various angles during training and perform feature matching at inference time for rapid pose estimation. Such model-based approaches augmented with synthetic training data [ 72 ] have shown state-of-the-art performance instance-level pose estimation. However, doing so requires either a CAD model or a small set of reference images annotated with the object poses, which becomes tedious as the number of unseen objects increases. On the other hand, there has been exciting progress in zero-shot, model-free methods over the past few years [ 69 , 71 ] which require no additional prior information, other than an object mask. BundleSDF [ 71 ] operates in a model-free manner by jointly optimizing a “neural object field” and the object poses by learning a 3D Signed Distance Field representation while concurrently running a global pose graph optimization. However, despite reporting near real-time pose optimization capabilities ( \sim 10Hz), the neural object field training is far from real-time 1 1 1 https://github.com/NVlabs/BundleSDF/issues/122 , which limits the average tracking frequency to \sim 0.4Hz. The significant computational overhead associated with training the neural object field hinders its applicability in live dynamic scenarios, where rapid pose updates are crucial. To address this limitation, we leverage Gaussian Splatting [ 25 , 22 ] which offers significantly better computational efficiency for real-time applications. We propose a novel method for online 6D object pose estimation through Gaussian Splatting, “6DOPE-GS”, that enables model-free, live object tracking and reconstruction. Building upon recent advances in using Gaussian Splatting for SLAM [ 24 ] , 6DOPE-GS jointly optimizes object poses from observed keyframes and reconstructs a 3D object model on the fly using incremental 2D Gaussian Splatting [ 22 ] . We propose several algorithmic enhancements to attain the required accuracy, efficiency, and training stability for live reconstruction and tracking. For accuracy, our method uses a novel dynamic keyframe selection mechanism to prioritize spatial coverage of the object and reconstruction confidence-based filtering to exclude keyframes with erroneous pose estimates. To maintain training stability and efficiency, we propose an adaptive Gaussian density control mechanism based on the opacity statistics of the Gaussians. Our contributions provide a significant speed-up in object pose estimation and tracking while maintaining high accuracy. In particular, we evaluate 6DOPE-GS on the HO3D and YCBInEOAT datasets and observe that it matches the state-of-the-art performance of competitive baselines while providing a 5 \times speedup. We also demonstrate the live, dynamic object tracking and reconstruction ability of 6DOPE-GS in a real-world setting. To the best of our knowledge, we are the first method to perform joint object tracking and Gaussian Splat reconstruction live at 3.5Hz from a single RGB-D camera. Our contributions are as follows: • We propose a novel method that effectively leverages 2D Gaussian Splatting for efficient and accurate model-free 6D object pose estimation and reconstruction. • We leverage the computationally efficient differentiable rendering of Gaussian Splatting to jointly optimize a 2D Gaussian Splatting-based “Gaussian Object Field” along with an object-centric pose graph of observed keyframes, that provides accurate, refined keyframe pose updates. • We propose a dynamic keyframe selection approach based on the spatial coverage of the set of keyframes and a reconstruction confidence-based filtering mechanism to exclude keyframes with erroneous pose estimates. • We incorporate a novel adaptive Gaussian density control mechanism based on opacity percentiles to filter out “unimportant” Gaussians, thereby improving training stability and computational efficiency. Figure 1 : Overview of our approach: 6DOPE-GS. Given a live input RGB-D video stream, we obtain object segmentation masks using SAM2 [ 50 ] on the incoming video frames. We then use LoFTR [ 58 ] , a transformer-based feature matching approach, to obtain pairwise correspondences between multiple views. We initialize a set of “keyframes” based on the density of matched features, for which we establish initial coarse pose estimates using RANSAC. To obtain refined pose updates for the keyframes, we use a 2D Gaussian Splatting-based “Gaussian Object Field” that is jointly optimized with the keyframe poses in a concurrent thread. We filter out erroneous keyframes for accurate pose refinement updates using a novel dynamic keyframe selection mechanism based on spatial coverage and reconstruction confidence. Moreover, we incorporate an opacity percentile-based adaptive density control mechanism to prune out inconsequential Gaussians, thus improving training stability and efficiency. Once the Gaussian Object Field is updated, it is temporarily frozen and the poses of keyframes that were filtered out are also updated. The object pose estimate at each timestep is then obtained by performing an online pose graph optimization using the incoming keyframe with the current set of keyframes. 2 Related Work 2.1 Object Pose Estimation and Tracking Instance-level 6D object pose estimation typically requires object CAD models and/or pretraining [ 19 , 18 , 27 , 64 , 48 , 30 , 66 , 70 , 63 , 4 ] . Such instance-level methods can be further categorized into correspondence-based [ 49 , 62 , 47 ] , template-based [ 60 , 8 ] , voting-based [ 31 , 19 , 41 , 18 , 64 ] , and regression-based [ 13 , 21 ] methods. For better generalization, some approaches use an object CAD model only at inference time [ 79 , 28 , 54 ] . Other methods [ 32 , 45 , 17 , 20 , 59