Category-Level 3D Correspondence in Camera Space via Morphable Object Priors
Leonhard Sommer, Artur Jesslen, Basavaraj Sunagad, Adam Kortylewski · 2026
Read Category-Level 3D Correspondence in Camera Space via Morphable Object Priors by Leonhard Sommer, Artur Jesslen, Basavaraj Sunagad, Adam Kortylewski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28257.txt.
Introduction
Sommer and colleagues learn category-level 3D point matches from single photos with no direct match labels. Their Morpheus method splits standard shape, warp, and pose into one shared shape prior, so steady 3D spots surface on their own. The method leads on their HouseCorr3D test set of home objects.
Catalog record
| CREATOR | Leonhard Sommer, Artur Jesslen, Basavaraj Sunagad, Adam Kortylewski |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82138 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.28257
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Category-Level 3D Correspondence in Camera Space via Morphable Object Priors Leonhard Sommer, Artur Jesslen, Basavaraj Sunagad, Adam Kortylewski Abstract Understanding 3D objects from images is fundamental to robotics and AR/VR applications. While recent work has made progress in category-level pose estimation, current representations fail to capture the fine-grained semantics needed for reasoning about object parts, functions, and interactions. In this work, we study category-level 3D correspondence in camera space —predicting, from a single image, 3D locations that remain consistent across instances within a category—and show that it can emerge without explicit correspondence supervision by learning a shared morphable object prior. To enable research in this direction, we introduce HouseCorr3D , the first large-scale benchmark for monocular category-level 3D correspondence with 178k images across 50 household object categories, 280 unique instances, and 3D keypoint annotations directly on CAD models. Crucially, HouseCorr3D provides amodal correspondence labels for occluded regions and explicit symmetry annotations, addressing key limitations of existing datasets. We further propose Morpheus , a method that learns morphable category-level shape priors by disentangling canonical shape, deformation, and object pose. Through this shared canonical grounding, semantically meaningful 3D correspondences in camera space emerge implicitly. These emerging 3D correspondences set a new state of the art on HouseCorr3D, demonstrating that semantic 3D object understanding can arise without direct correspondence supervision. Data and code: /GenIntel/HouseCorr3D . 1 Introduction Figure 1 : Monocular Category-level 3D Correspondence. We predict semantically consistent 3D keypoint locations across different instances of the same category from single RGB-D images. Our morphable priors enable establishing correspondences (shown with matching colors) that remain semantically aligned despite large shape variations, enabling fine-grained object understanding beyond traditional pose estimation and 2D semantic correspondence. Understanding objects in 3D from images is a long-standing challenge in computer vision, with applications in robotics, augmented reality (AR), and virtual reality (VR). Traditional 3D object understanding has primarily focused on pose estimation, object detection, or 3D reconstruction. However, current approaches fail to capture the fine-grained semantics needed for reasoning about object parts, their functions, and how they can be manipulated or interacted with. A key step toward richer understanding is to establish semantic correspondences – estimating which points on different objects represent the same functional part. In 2D, this problem has driven extensive research [ 27 , 37 , 16 , 29 , 26 ] , enabling applications like image matching, retrieval, and style transfer. Yet, 2D correspondences are inherently limited by viewpoint dependence, occlusion, and symmetry ambiguities. We therefore propose to move beyond 2D, and towards the prediction of semantically aligned 3D locations that remain consistent across all instances of a category (as illustrated in Fig. 1 ). Unlike prior work that maps pixels into normalized canonical spaces [ 19 , 45 ] , we propose to establish correspondences directly in 3D camera space, resolving fundamental ambiguities that arise in image-space matching due to occlusion, viewpoint change, and scale variation. Formally, we define this novel task as follows: Monocular Category-level 3D correspondence: Given two query and target RGB-D images \mathrm{I}^{q} and \mathrm{I}^{t} of objects from the same category, and a query 3D point x^{q}\!\in\!\mathbb{R}^{3} in the camera space of \mathrm{I}^{q} , the task is to predict the 3D point x^{t}\!\in\!\mathbb{R}^{3} in \mathrm{I}^{t} camera space that corresponds to the same semantic point. Intuitively, the task asks: if we select a semantic part on one object, where does the same part lie on another instance of the category? Our approach answers this question by mediating correspondence through a shared deformable template. An overview of this camera-space correspondence setup is illustrated in Fig. 3 a . Unfortunately, existing benchmarks such as NOCS-Real275 [ 45 ] , Wild6D [ 7 ] , OmniNOCS [ 19 ] , and Omni6DPose [ 60 ] only provide pose annotations, segmentation, and depth, but lack category-level 3D correspondences . To address this gap, we introduce HouseCorr3D , a large-scale benchmark for monocular category-level 3D correspondence in camera space. HouseCorr3D covers 50 everyday object categories with 178k images and 280 unique object instances, each annotated with semantic 3D keypoints directly on CAD models that project consistently across all views. Crucially, our annotations include amodal correspondences —correspondences for object parts that are occluded or not visible in the image. This capability is inspired by human reasoning [ 57 ] , where we naturally infer the complete 3D structure of objects even under occlusion, and is essential for robotic manipulation where planning grasps and interactions requires understanding the full spatial extent of objects [ 53 ] , not just visible surfaces. We also explicitly support object symmetries, ensuring symmetric objects have multiple valid correspondences and avoiding unfair penalization of symmetry-equivalent predictions. Together, these properties address fundamental limitations of pose-focused datasets and, for the first time, enable quantitative evaluation of category-level 3D correspondence from single images. On HouseCorr3D, we show that monocular category-level 3D correspondence can emerge without explicit correspondence supervision by constraining object instances through a shared deformable representation. To this end, we propose Morpheus , a framework that learns morphable category-level shape priors to produce semantically consistent 3D correspondences directly in camera space. Instead of relying on a fixed representation, Morpheus learns a deformable 3D template for each category that adapts to instance-specific shape variations while preserving correspondences. During training, our method jointly optimizes a 3D morphable prior, instance-specific shape deformations, and their 2D projection consistency. At inference, given a single RGB-D image, Morpheus predicts both the object’s 3D shape in camera space and its semantically aligned keypoints, enabling correspondence evaluation without pose normalization. In summary, our contributions are as follows: (i) We identify monocular category-level 3D correspondence in camera space as a key next step beyond pose-centric representations toward semantically aligned 3D understanding. (ii) We introduce HouseCorr3D , the first large-scale benchmark for category-level 3D correspondence, comprising 178k images across 50 household categories and 280 instances, with mesh-based keypoint annotations, amodal correspondences, and explicit symmetry labels. (iii) We propose Morpheus , a framework that learns morphable category-level shape priors to establish semantically consistent 3D correspondences directly in camera space. (iv) We demonstrate that Morpheus substantially outperforms existing baselines on HouseCorr3D, establishing a new paradigm for correspondence-level 3D object understanding . 2 Related work 2D Semantic Correspondence. 2D correspondence has advanced from local descriptors and dense flows ( e.g . , SIFT [ 25 ] , DAISY [ 42 ] , SIFT Flow [ 22 ] , DeepFlow [ 47 ] ) to transformer-based self-supervised features [ 3 , 63 , 32 , 61 ] , which exhibit emergent semantic alignment and achieve strong results on benchmarks like SPair-71K, PF-PASCAL, and TSS [ 27 , 13 , 21 ] . Dedicated matchers such as LoFTR, COTR, DiffMatch [ 37 , 16 , 29 ] , and spherical-map approaches [ 26 , 6 ] further improve dense matching. While highly effective, these approaches remain limited to the image domain and do not predict 3D canonical coordinates or enforce semantic consistency across instances in 3D space. 3D Keypoint and Correspondence Methods. Prior work explored correspondence mapping in the 3D domain through keypoint detection and surface mapping. KeypointNet [ 59 ] introduced a large-scale dataset for learning category-consistent 3D keypoints, while others [ 15 , 56 ] leverage keypoints for cage-based