WALDO: Where Unseen Model-based 6D Pose Estimation Meets Occlusion

Sajjad Pakdamansavoji, Yintao Ma, Amir Rasouli, Tongtong CaoHuawei Technologies Canada, [email protected] · 2025

Read WALDO: Where Unseen Model-based 6D Pose Estimation Meets Occlusion by Sajjad Pakdamansavoji, Yintao Ma, Amir Rasouli, Tongtong CaoHuawei Technologies Canada, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-15874.txt.

Introduction

Sajjad Pakdamansavoji, Yintao Ma, Amir Rasouli, and Tongtong Cao present WALDO, a method for estimating the full pose of objects a system has never seen, even when they are partly hidden. Pose means position plus orientation in three dimensions, vital for robotics and augmented reality. Pipelines usually detect and segment the object, propose a pose, then refine it, but occlusion corrupts the early steps and errors cascade.

Four extensions address that weakness. Sampling focuses on visible regions, several ranked pose hypotheses are kept instead of one brittle guess, refinement iterates to sharpen accuracy, and training augmentations emphasize occlusion. A new visibility-weighted metric reduces bias in scoring under occlusion, and extensive tests support the approach.

Catalog record

CREATORSajjad Pakdamansavoji, Yintao Ma, Amir Rasouli, Tongtong CaoHuawei Technologies Canada, [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-15874

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

WALDO: Where Unseen Model-based 6D Pose Estimation Meets Occlusion Sajjad Pakdamansavoji, Yintao Ma, Amir Rasouli, Tongtong CaoHuawei Technologies Canada, [email protected] Abstract Accurate 6D object pose estimation is vital for robotics, augmented reality, and scene understanding. For seen objects, high accuracy is often attainable via per-object fine-tuning but generalizing to unseen objects remains a challenge. To address this problem, past arts assume access to CAD models at test time and typically follow a multi-stage pipeline to estimate poses: detect and segment the object, propose an initial pose, and then refine it. Under occlusion, however, the early-stage of such pipelines are prone to errors, which can propagate through the sequential processing, and consequently degrade the performance. To remedy this shortcoming, we propose four novel extensions to model-based 6D pose estimation methods: (i) a dynamic non-uniform dense sampling strategy that focuses computation on visible regions, reducing occlusion-induced errors; (ii) a multi-hypothesis inference mechanism that retains several confidence-ranked pose candidates, mitigating brittle single-path failures; (iii) iterative refinement to progressively improve pose accuracy; and (iv) series of occlusion-focused training augmentations that strengthen robustness and generalization. Furthermore, we propose a new weighted by visibility metric for evaluation under occlusion to minimize the bias in the existing protocols. Via extensive empirical evaluations, we show that our proposed approach achieves more than 5% improvement in accuracy on ICBIN and more than 2% on BOP dataset benchmarks, while achieving ≈ 3 × \approx 3\times faster inference. 1 Introduction Accurate 6D object pose estimation is a fundamental problem in various domains, such as robotics, augmented reality, and scene understanding. Over the past decade a wide range of methods have been proposed, including model-free methods that only rely on images or videos of the objects [ 19 ] or model-based methods that resort to using CAD models for more generalability to unseen objects [ 28 ] . These innovations have been supported by a wide range of techniques, including synthetic data generation, including large-scale 3D asset repositories [ 6 , 14 , 9 ] , photorealistic rendering pipelines [ 46 , 11 ] , and neural architectural innovations [ 39 , 54 ] . These methods enabled large-scale pretraining, leading to better generalization to out-of-distribution (OOD) scenarios as well as narrowing the sim-to-real gap. Figure 1 : Scatter plot comparing unbiased average recall (UAR) versus inference time for multiple methods on the BOP-Core* dataset. WALDO performs better compared to past arts on both metrics. Despite the substantial progress, accurate 6D pose estimation under practical conditions involving clutter and occlusion remains an open problem, in particular for unseen objects. In the unseen object setting state-of-the-art methods assume a CAD model at the test time and typically follow a detect-segment-pose-refine pipeline. However, under occlusion, errors introduced in earlier stage would potentially accumulate and degrade pose estimation accuracy. To remedy this problem, several solutions have been proposed, including domain randomization [ 26 ] , additional modalities incorporation [ 5 ] , local-feature extraction and search [ 51 ] , and leveraging large foundation models [ 3 ] . However, these approaches typically address occlusion only indirectly, are computationally expensive, and often depend on strong assumptions about the environment, limiting their practical use in previously unexplored scenes. Besides the limitations of the existing approaches, evaluation protocols are not adequate for assessment under occlusion. The existing benchmarks [ 19 ] lack systematic scene variation and their metrics do not account for the occlusion distribution in the data. Hence, the reported results are biased toward high-visibility cases, inflating reported performance and obscuring the true impact of occlusion. In this work, we address the problem of 6D pose estimation under occlusion by coupling inference-time strategies that focus on visible regions of the object with training-time augmentations that simulate partial visibility. To navigate pose uncertainty, we maintain multiple initial hypotheses and iteratively refine them, yielding higher accuracy. Finally, we propose an occlusion-aware variant of standard evaluation metrics that provides unbiased performance estimates across visibility levels. In summary, the contributions of our work are: • We address occlusion by estimating occlusion probabilities while forming the initial pose hypothesis and then use them to prioritize visible regions during refinement. • To manage pose uncertainty, we generate multiple initial hypotheses, refine them iteratively, and select the final pose based on its post-refinement confidence score. • We propose training-time augmentations specifically designed to simulate occlusion and improve robustness to partial visibility. • We propose occlusion-aware evaluation metrics for 2D detection, segmentation, and 6D pose estimation that provide unbiased performance across visibility levels. 2 Related Work 6D Pose Estimation. models can be categorized based on wether the objects are seen during training or are unseen . For seen objects, early systems [ 40 , 22 ] were trained per-instance to learn extracting stable 2D–3D cues, such as keypoints [ 37 ] , view/rotation labels [ 52 ] , dense surface codes [ 15 ] , and geometry-guided predictors [ 49 , 42 , 53 ] . using exact CAD models and per-object fine-tuning, these frameworks can reach high accuracy even under significant occlusion because correspondence supervision tightly binds image evidence to the specific mesh [ 16 ] . Generalizing to unseen instances at test time is notably harder, especially under occlusion because overfitting to specific shapes or contexts does not transfer. Model-free methods attempt to remedy this problem by building an object-specific reference bank using only a few views of the objects and estimate their pose at run-time by local feature matching and refinement [ 43 , 31 ] . However, using an incomplete reference model can lead to failure under occlusion due to sparse observation and weak matching results. To remedy this issue, model-based resort to using object CAD models. Some use a single-pass 6D pose estimation at inference, using direct regression/classification [ 45 ] or implicit pose-code retrieval [ 44 ] . This approach, however, comes at a cost. Due to the lack of a self-correction mechanism, error from imperfect detection or occluder confusion cannot be recovered, therefore accuracy degrades as occlusion increases. Two-stage methods solve this problem by, first deriving an initial pose and then refining it. For instance, the model in [ 25 ] predicts the correspondences between keypoints or dense object coordinates and estimates an initial pose via solving a Perspective-n-Point (PnP) [ 55 ] problem. Other methods extend this by adding a refinement module to align a rendered view to the image (render-and-compare) [ 27 ] or fuse depth with learned features [ 48 ] to reduce residual error. Such architectures allow self-correction under occlusion where the refiner can repair mislabeled evidence using the CAD prior. However, because the pipeline is multi-stage, weak initial hypothesis can still propagate and limit final accuracy unless hypothesis diversity and scoring are carefully designed [ 24 ] . In this work, we adopt a model-based approach and use a novel sampling method to mitigate the error propagation problem. Figure 2 : Overview of our 6D pose estimation framework. The observed RGB–D image and the rendered RGB–D template are first sent to MUSE to obtain bounding boxes and masks. Using these, the inputs are processed and passed through a feature extractor to compute embeddings and their corresponding point clouds. A uniform coarse sample

Read the complete text