Cov2Pose : Leveraging Spatial Covariance for Direct Manifold-aware 6-DoF Object Pose Estimation

Nassim Ali Ousalah, Peyman Rostami, Vincent Gaudillière, Emmanuel Koumandakis, Anis Kacem, Enjie Ghorbel, Djamila Aouada, Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France, Infinite Orbits, Cristal Laboratory, ENSI, University of Manouba {nassim.aliousalah,peyman.rostami,anis.kacem,djamila.aouada}@[email protected], [email protected], [email protected] · 2026

Read Cov2Pose : Leveraging Spatial Covariance for Direct Manifold-aware 6-DoF Object Pose Estimation by Nassim Ali Ousalah, Peyman Rostami, Vincent Gaudillière, Emmanuel Koumandakis, Anis Kacem, Enjie Ghorbel, Djamila Aouada, Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France, Infinite Orbits, Cristal Laboratory, ENSI, University of Manouba {nassim.aliousalah,peyman.rostami,anis.kacem,djamila.aouada}@[email protected], [email protected], [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19961.txt.

Introduction

Ousalah and co-authors estimate an object's full 6D pose from one RGB image, directly. Direct methods are fast but often less accurate than keypoint methods. They also tend to pool features globally, losing spatial detail, and use pose codes with jumps. Cov2Pose instead pools feature maps into a covariance matrix, a symmetric positive definite object. The pose is coded through a Cholesky factor of such a matrix. A manifold aware head then regresses the pose end to end. Tests, including partial occlusion, favor the new design.

Catalog record

CREATORNassim Ali Ousalah, Peyman Rostami, Vincent Gaudillière, Emmanuel Koumandakis, Anis Kacem, Enjie Ghorbel, Djamila Aouada, Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France, Infinite Orbits, Cristal Laboratory, ENSI, University of Manouba {nassim.aliousalah,peyman.rostami,anis.kacem,djamila.aouada}@[email protected], [email protected], [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64793
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19961

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cov2Pose : Leveraging Spatial Covariance for Direct Manifold-aware 6-DoF Object Pose Estimation Nassim Ali Ousalah, Peyman Rostami, Vincent Gaudillière, Emmanuel Koumandakis, Anis Kacem, Enjie Ghorbel, Djamila Aouada, Université de Lorraine, CNRS, Inria, LORIA, F-54000 Nancy, France, Infinite Orbits, Cristal Laboratory, ENSI, University of Manouba {nassim.aliousalah,peyman.rostami,anis.kacem,djamila.aouada}@[email protected], [email protected], [email protected] Abstract In this paper, we address the problem of 6-DoF object pose estimation from a single RGB image. Indirect methods that typically predict intermediate 2D keypoints, followed by a Perspective- n n -Point solver, have shown great performance. Direct approaches, which regress the pose in an end-to-end manner, are usually computationally more efficient but less accurate. However, direct pose regression heads rely on globally pooled features, ignoring spatial second-order statistics despite their informativeness in pose prediction. They also predict, in most cases, discontinuous pose representations that lack robustness. Herein, we therefore propose a covariance-pooled representation that encodes convolutional feature distributions as a symmetric positive definite (SPD) matrix. Moreover, we propose a novel pose encoding in the form of an SPD matrix via its Cholesky decomposition. Pose is then regressed in an end-to-end manner with a manifold-aware network head, taking into account the Riemannian geometry of SPD matrices. Experiments and ablations consistently demonstrate the relevance of second-order pooling and continuous representations for direct pose regression, including under partial occlusion. 1 Introduction Estimating the six-degree-of-freedom (6-DoF) pose of an object from an RGB image is a core challenge in computer vision 44 ; 43 ; 33 ; 74 ; 48 ; 3 ; 51 ; 4 due to its wide range of applications in robotic manipulation 69 ; 31 , augmented and virtual reality 85 , autonomous navigation 11 ; 64 , and industrial inspection 37 ; 53 ; 24 . Given an image, the goal is to predict the pose P\in\mathrm{SE}(3) of the object of interest relative to the camera that captured the image, where \mathrm{SE}(3) denotes the Special Euclidean Group of \mathbb{R}^{3} . Figure 1 : Pose–representation alignment . Pairs of images 2 2 2 Training images from the 002_master_chef_can class of the YCB-V 87 dataset. We use an EfficientNet-B6 73 pretrained on ImageNet 15 with no prior training on pose estimation. are sorted according to the geodesic distances in \mathrm{SE}(3) between their corresponding ground-truth poses (x-axis). The x-axis is then partitioned into bins. For each bin, we plot: (i) the average Log–Euclidean distance between spatial covariances of image pairs within this bin (red) and (ii) the average Cosine distance between flattened features of image pairs within the bin (blue). Covariance distances increase with pose distance, whereas flattened feature distances remain nearly constant, indicating that spatial covariance is more correlated with the pose. Thanks to the tremendous developments made in deep learning, numerous approaches have leveraged Deep Neural Networks (DNNs) 88 ; 36 ; 85 ; 50 either directly or indirectly to estimate the 6-DoF object pose. Indirect methods mainly predict intermediate representations such as keypoints 66 ; 88 ; 86 ; 48 or dense object coordinates 89 ; 80 that are used to solve a Perspective- n -Point (P n P) problem or adopt a render-and-compare approach that optimizes photometric, silhouette, or differentiable rendering losses to align a rendered model view with the input image 45 ; 54 ; 42 ; 46 . Direct methods, on the other hand, regress rotation and translation in an end-to-end manner using a neural architecture 47 ; 87 ; 79 ; 16 ; 77 ; 13 ; 52 . While indirect methods have been demonstrated to be more accurate, they suffer from a high calculation cost induced by the iterative computation they incur 45 . In contrast, direct methods are usually more efficient despite being less precise 79 ; 13 ; 12 , thereby suggesting their practical relevance in real-time applications. In this paper, we argue that we can improve the accuracy of direct methods by incorporating the spatial second-order statistics of the input image within the DNN-based pose estimator. In fact, deep pose regressors typically rely solely on first-order statistics of feature activations. Nonetheless, second-order moments such as covariance matrices can capture pairwise co-activations that average or max pooling operations discard, thereby providing information about the spatial variability within an image. We hypothesize that such information is highly relevant for inferring the pose. This hypothesis is confirmed by the experiment reported in Figure 2 . This figure shows that pairwise distances between spatial covariances of DNN features increase with ground-truth \mathrm{SE(3)} pose distance, whereas distances between flattened features remain nearly constant. This suggests that spatial covariance is strongly correlated with pose and provides, as a result, a more suitable representation for direct pose regression. Concretely, we propose to make use of second-order pooling, which has shown its utility in several deep learning-based computer vision tasks such as classification, facial expression recognition, object detection, and action recognition 34 ; 25 ; 22 ; 21 ; 81 ; 82 ; 10 ; 2 . This pooling strategy returns feature covariance matrices that are known to be symmetric positive-definite (SPD), ensuring the exploitation of statistical information beyond first-order within a DNN, while preserving spatial awareness. To the best of our knowledge, we are the first to explore spatial covariance representations for end-to-end pose regression. Our framework, termed Cov2Pose , is an end-to-end trainable pipeline that respects the geometry of the SPD manifold where covariance matrices reside. Moreover, we introduce within Cov2Pose a one-to-one continuous mapping from the SPD space to \mathrm{SE}(3) via a differentiable Cholesky-based encoding without breaking the chain rule, allowing the direct prediction of a continuous 6-DoF pose representation. In particular, the pose is predicted under the shape of a continuous 6D rotation parametrization coupled with a 3D translation. This chosen rotation representation has been shown to be more suitable for rotation learning 90 . Experiments performed on several object pose benchmarks show the superiority of Cov2Pose compared to previous direct methods, demonstrating the relevance of the proposed principled framework. In summary, our main contributions can be summarized as follows, • The first covariance-based deep learning framework for RGB-based end-to-end 6-DoF object pose regression, which leverages the spatial covariance of backbone features to encode higher-order statistics. • A manifold-aware training pipeline that applies geometry-preserving dimensionality reduction through Bilinear Mapping layers (BiMap) 32 to a compact SPD representation. • A fully differentiable, CAD-free, one-to-one, and continuous pose regressor that maps the latent SPD matrix to a continuous 6D rotation representation and translation via a differentiable Cholesky decomposition. • Extensive experiments on three object pose benchmarks, namely, LineMOD 26 , Occ-LineMOD 7 and YCB-Video 87 , showing that Cov2Pose achieves state-of-the-art results as compared to direct regression methods. The remainder of the paper is organized as follows: Section 2 presents the related work on 6-DoF object pose estimation and the background. Section 3 formulates the problem of direct pose regression. Section 4 describes the proposed framework entitled Cov2Pose . Section 5 reports the conducted experiments and discusses the obtained results. Finally, Section 6 concludes this work, highlighting potential future directions. 2 Related Work and Background In this section, we first review recent literature on RGB-based 6-DoF object pose estimation, then discuss the use of covariance representations in vision tasks. We also provide preliminaries on the SPD manifold, where

Read the complete text