CRISP : Object Pose and Shape Estimation with Test-Time Adaptation

Jingnan Shi, Rajat Talak, Harry Zhang, David Jin, Luca Carlone · 2024

Read CRISP : Object Pose and Shape Estimation with Test-Time Adaptation by Jingnan Shi, Rajat Talak, Harry Zhang, David Jin, Luca Carlone (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-01052.txt.

Introduction

Shi, Talak, Zhang, Jin, and Carlone present CRISP, a system that finds the pose and shape of an object from one color and depth image. Their paper is from two thousand twenty-four. CRISP does not need to know the object's class in advance. It pairs a shape model with a network that finds points in a fixed pose frame.

A second part fixes pose and shape errors that come from a gap in the data. A third part lets CRISP train itself at test time, using its own checked fixes as labels. Tests on three data sets show strong results. The self-training step can bridge a large gap, and CRISP can work on objects it has not seen.

Catalog record

CREATORJingnan Shi, Rajat Talak, Harry Zhang, David Jin, Luca Carlone
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE91 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-01052

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CRISP : Object Pose and Shape Estimation with Test-Time Adaptation Jingnan Shi, Rajat Talak, Harry Zhang, David Jin, Luca Carlone Abstract We consider the problem of estimating object pose and shape from an RGB-D image. Our first contribution is to introduce CRISP , a category-agnostic object pose and shape estimation pipeline. The pipeline implements an encoder-decoder model for shape estimation. It uses FiLM-conditioning for implicit shape reconstruction and a DPT-based network for estimating pose-normalized points for pose estimation. As a second contribution, we propose an optimization-based pose and shape corrector that can correct estimation errors caused by a domain gap. Observing that the shape decoder is well behaved in the convex hull of known shapes, we approximate the shape decoder with an active shape model, and show that this reduces the shape correction problem to a constrained linear least squares problem, which can be solved efficiently by an interior point algorithm. Third, we introduce a self-training pipeline to perform self-supervised domain adaptation of CRISP . The self-training is based on a correct-and-certify approach, which leverages the corrector to generate pseudo-labels at test time, and uses them to self-train CRISP . We demonstrate CRISP (and the self-training) on YCBV, SPE3R, and NOCS datasets. CRISP shows high performance on all the datasets. Moreover, our self-training is capable of bridging a large domain gap. Finally, CRISP also shows an ability to generalize to unseen objects. Code and pre-trained models will be available on the project webpage. 1 1 1 https://web.mit.edu/sparklab/research/crisp_object_pose_shape/ 1 Introduction Figure 1 : We introduce CRISP , a category-agnostic object pose and shape estimation pipeline, and a test-time adaptive self-training method CRISP - ST to bridge domain gaps. Top: Qualitative examples of CRISP on the YCBV dataset [ 45 ] . Bottom: Qualitative examples of CRISP on the SPE3R dataset [ 27 ] . Accurately estimating the geometry of objects is necessary in many vision applications ranging from augmented reality [ 48 ] to robotics [ 10 ] , and autonomous docking in space [ 4 ] . While significant progress has been made in solving object perception [ 45 , 42 , 47 ] , real-world deployments remain challenging due to lack of generalizable models and in-domain training data [ 34 ] . Existing approaches to object perception tasks focus on instance-level 6D pose estimation [ 15 , 16 ] , category-level object 6D pose and shape estimation [ 39 , 47 ] , 3D bounding box estimation [ 2 , 23 ] , and more recently category-agnostic object pose and shape estimation [ 22 ] . Approaches for the first two tasks usually require object CAD model or category-level priors at inference time. On the other hand, 3D bounding box may be too coarse a representation to be useful in many applications ( e.g . autonomous space debris removal [ 27 ] ). Recent work on conditional diffusion models have shown promise in single-view 3D reconstruction [ 13 , 20 ] . However, their inference time remains prohibitively high for real-time operation [ 20 ] . In addition, the domain gap issue, where the distribution of the test data differs from training, remains a major concern. Developing an object perception model is not enough as a domain gap can render its estimates completely useless, or worse still, hazardous in safety-critical applications [ 34 ] . Therefore, in conjunction to the object perception model, it is necessary to develop methods that can bridge any large domain gap and work well at test-time. Towards addressing these issues, this paper presents three main contributions: • We introduce CRISP , an object pose and shape estimation pipeline. CRISP combines a pre-trained vision transformer (ViT) backbone with a dense prediction transformer (DPT) and feature-wise linear modulation (FiLM) conditiong to estimate the 6D pose and shape of the 3D object from a single RGB-D image [ 33 , 31 ] . CRISP is category-agnostic ( i.e . , it does not require knowledge of the object category at test time). • We propose an optimization-based pose and shape corrector that can correct estimation errors. The corrector is a bi-level optimization problem and we use block coordinate descent to solve it. We approximate the shape decoder in CRISP by an active shape model, and show that (i) this is a reasonable approximation, and (ii) doing so turns it into a constrained linear least squares problem, which can be solved efficiently using interior-point methods and yields just as good shapes as the trained decoder. • We adapt a correct-and-certify approach to self-train CRISP and bridge any large domain gap. During self-training, we use the corrector to correct for pose and shape estimation errors. Then, we assert the quality of the output of the corrector using an observable correctness certificate inspired by [ 36 ] , and create pseudo-labels using the estimates that pass the certificate check. Finally, we train the model on these pseudo-labels with standard stochastic gradient descent. Contrary to [ 30 , 22 ] , we do not need access to synthetic data during self-training. We demonstrate CRISP on the YCBV, SPE3R, and NOCS datasets. CRISP shows high performance on all the datasets. Moreover, our self-training is able to bridge a large domain gap.Finally, we observe that CRISP is able to generalize and estimate pose and shape of objects unseen during training, and is fast for real-time inference. 2 Related Work Object Pose and Shape Estimation. The task of object pose and shape estimation is to recover the 3D poses and shape of an object under observation. This is different from the task of instance-level pose estimation, which usually assumes object CAD models are provided during test-time inference [ 43 , 41 , 45 , 15 , 16 , 18 , 7 ] . Category-level approaches have been proposed, which aim to simultaneously estimate the shape and pose of the objects, despite large intra-class shape variations. One common paradigm in category-level pose and shape estimation is to first regress some intermediate representations, and solve for poses and shapes. Pavlakos et al. [ 29 ] use a stacked hourglass neural network [ 24 ] for 2D semantic keypoint detection. Wang et al. propose Normalized Object Coordinates (NOCS) to represent the objects as well as use them to recover pose and shape [ 42 ] . Many recent works adopt category-specific shape priors, which are deformed to match the observation during inference. Tian et al. [ 39 ] propose to learn the category mean shapes from training data, and then use them for pose and shape estimation. Ze and Wang [ 47 ] propose RePoNet, which trains the pose and shape branches using differentiable rendering. However, the use of category-level priors limit the scalability of these approaches. Recently, there have been a number of works using neural implicit fields for object pose and shape estimation [ 22 , 30 , 12 ] . Peng et al. [ 30 ] combine PointNet [ 32 ] with a DeepSDF decoder [ 26 ] for representing shapes. Irshad et al. [ 12 ] learn a disentangled implicit shape and apperance model for joint appearance, shape and pose optimization. Comparing with conditional diffuion models that are prevalent in the shape reconstruction

Read the complete text