ESCAPE: Equivariant Shape Completion via Anchor Point Encoding

Burak Bekci, Nassir Navab, Federico Tombari, Mahdi Saleh · 2024

Read ESCAPE: Equivariant Shape Completion via Anchor Point Encoding by Burak Bekci, Nassir Navab, Federico Tombari, Mahdi Saleh (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00952.txt.

Introduction

Burak Bekci, Nassir Navab, Federico Tombari, and Mahdi Saleh present ESCAPE, a method that completes the missing parts of a scanned three-dimensional object no matter how it is rotated. Most completion methods assume a known pose or fixed coordinates, so they falter when objects appear at arbitrary angles. Real scans rarely cooperate.

Their trick is to pick anchor points on the shape and describe every point by its distances to all anchors, a code that turns with the object. A transformer reads and writes these distance codes, and an optimization step rebuilds the shape. Tests show accurate, rotation-consistent completions across arbitrary rotations and shifts, without a separate pose estimation step.

Catalog record

CREATORBurak Bekci, Nassir Navab, Federico Tombari, Mahdi Saleh
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00952

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ESCAPE: Equivariant Shape Completion via Anchor Point Encoding Burak Bekci, Nassir Navab, Federico Tombari, Mahdi Saleh Abstract Shape completion, a crucial task in 3D computer vision, involves predicting and filling the missing regions of scanned or partially observed objects. Current methods expect known pose or canonical coordinates and do not perform well under varying rotations, limiting their real-world applicability. We introduce ESCAPE (Equivariant Shape Completion via Anchor Point Encoding), a novel framework designed to achieve rotation-equivariant shape completion. Our approach employs a distinctive encoding strategy by selecting anchor points from a shape and representing all points as a distance to all anchor points. This enables the model to capture a consistent, rotation-equivariant understanding of the object’s geometry. ESCAPE leverages a transformer architecture to encode and decode the distance transformations, ensuring that generated shape completions remain accurate and equivariant under rotational transformations. Subsequently, we perform optimization to calculate the predicted shapes from the encodings. Experimental evaluations demonstrate that ESCAPE achieves robust, high-quality reconstructions across arbitrary rotations and translations, showcasing its effectiveness in real-world applications without additional pose estimation modules. 1 Introduction 3D perception has been dependent on canonical orientations, causing significant challenges in dynamic environments, such as robotic manipulation or real-time object recognition, where objects interact from varying viewpoints. Traditional methods, like voxel-based approaches and early point cloud networks such as PointNet qi2017pointnet and PointNet++ qi2017pointnet++ , have laid a strong foundation but cannot inherently handle rotational variance. Recent methods have introduced improvements by leveraging attention mechanisms and hierarchical processing capabilities but still rely on orientation normalization or data augmentation techniques to perform robustly with rotational changes, which do not fully resolve the inherent challenges of varying viewpoints. Figure 1 : Existing shape completion methods (Top) use conventional canonical coordinates and perform poorly under rotation changes and unknown canonical reference. Using our anchor point encoding (Below), we manage to consistently complete the shape with arbitrary rotation in non-canonical coordinates. Despite progress in the field, existing shape completion techniques struggle to maintain robustness and accuracy under arbitrary rotations. This notably limits their practicality in real-world applications where objects can appear in any orientation. Recent methods, including transformer-based models like SnowflakeNet xiang2021snowflakenet and SeedFormer zhou2022seedformer , have introduced improvements by leveraging the attention mechanisms, progressively growing points, and hierarchical processing capabilities of transformers. However, these models expect partial shapes in aligned orientation across a category and only learn completion in canonical coordinates, which do not guarantee performance consistency in varying viewpoints or unknown local coordinates. Recent advances in rotation-invariant descriptors, particularly through Vector Neurons and SE(3)-equivariant networks, have shown promise in tasks like registration and classification. However, these approaches face limitations when applied to shape completion: their layerwise transformations can compound errors through deep networks ( \mathcal{O}(\alpha^{L}) for L layers), and their equivariant constraints may sacrifice geometric information critical for completing missing regions. While these methods achieve equivariance, they don’t provide guarantees on geometric completeness - two shapes with identical vector features aren’t necessarily isometric. To address these challenges, we introduce ESCAPE, a novel framework specifically designed for rotation-equivariant shape completion. Our key insight is representing shapes through distances to dynamically selected anchor points, which provides several theoretical advantages: constant error bounds ( \mathcal{O}(1) ) independent of network depth, guaranteed geometric completeness (identical distance encodings imply isometric shapes), and provable reconstruction uniqueness with sufficient anchor points. ESCAPE processes this representation through a transformer architecture, leveraging attention mechanisms to predict distances to anchor points for the complete shape while maintaining rotational equivariance. This approach not only preserves geometric fidelity crucial for completion but also offers practical benefits - simpler optimization without specialized equivariant layers and natural handling of varying partiality through adaptive anchor selection. We propose a complete pipeline to generate, encode, decode, and interpret rotation equivariant features for non-canonical 3D shape completion. In summary, our contributions are threefold: • We introduce ESCAPE, a novel rotation-equivariant 3D encoding strategy using high-curvature anchor points, enabling robust shape description and reconstruction. • We develop a transformer-based architecture that leverages our equivariant encoding to generate the completed point cloud, maintaining consistency across varying object orientations and partial inputs. • We present the an end-to-end rotation-equivariant shape completion method, demonstrating robust performance under arbitrary rotations and in the absence of canonical object coordinates. • We establish and evaluate a challenging real-world shape completion benchmark using the OmniObject dataset wu2023omniobject3d , featuring partial point clouds with diverse geometries and arbitrary poses, bringing shape completion works to real-world data. 2 Related Works 2.1 Point Cloud Processing Point clouds, representing unordered and sparse 3D data, challenge conventional CNNs. Voxel-based methods like VoxNet maturana2015voxnet and VoxelNet zhou2018voxelnet convert point clouds into regular grids for 3D CNN processing but are computationally intensive. PointNet qi2017pointnet and PointNet++ qi2017pointnet++ address unordered sets with MLPs and hierarchical grouping. PointNet creates a permutation-invariant structure with pointwise MLPs, while PointNet++ adds hierarchical layers and furthest point sampling, increasing computational complexity. These methods are often combined with voxel-based approaches in large-scale applications shi2020pv ; zhou2018voxelnet . Graph neural networks (GNNs) like PointCNN li2018pointcnn and EdgeConv wang2019dynamic capture local geometric features through spatial relationships and dynamic graph construction. Recent advancements integrate edge-based attention for improved efficiency velivckovic2017graph ; vaswani2017attention . Transformers and attention mechanisms have also been applied to point clouds, with Graph Attention Networks (GATs) velivckovic2017graph using self-attentional layers and the Transformer architecture vaswani2017attention introducing multi-head attention for robust feature aggregation. Set Transformer lee2019set and Perceiver jaegle2021perceiver adapt attention mechanisms for 3D data. Point cloud-specific transformers, such as PCT guo2020pct and Point Transformer zhao2021point , apply self-attention to neighborhood points, though they can be computationally expensive. 2.2 Rotation-Invariant Encodings Handcrafted rotation-invariant descriptors have been widely explored in 3D by researchers before the popularity of deep neural networks. To guarantee invariance under rotations, many handcrafted local descriptors rusu2008aligning ; rusu2009fast ; tombari2010unique ; guo2013rotational rely on an estimated local reference frame (LRF), which is typically based on the covariance analysis of the local surface, to transform local patches to a canonical representation. The major drawback of LRF is its non-uniqueness, making the constructed rotational invariance fragile and sensitive to noise. Consequently, attention has shifted to LRF-free approaches drost2010model . These methods focus on mining the rotation-invariant components of local surfaces to represent the local geometry. For instance, PPF drost2010model , PFH rusu2008aligning , and FPFH rusu2009fast encode the geometry of the local surface using histograms of pairwise geometrical properties. Early work by Osada et al. osada2002shape introduced shape distributions using distances between random surface points as rotation-invariant descriptors. Despite being rotation-invariant by design, these handcrafted descriptors are often inadequate for complex geometry and noisy data. Recently, many deep learning-based methods have aimed to learn rotation-invariant descriptors. PPF-FoldNet deng2018ppf encodes PPF patches into embeddings, using a FoldingNet yang2018foldingnet decoder to reconstruct the input, enabling correspondences from the rotation-invariant embeddings. SpinNet ao2021spinnet and Graphite saleh2020graphite ; saleh2022bending align local patches to defined axes before learning descriptors. However, these methods are limited by their locality, as descriptors are learned only from the local region, making them less distinctive. YOHO xu2021you introduces a rotation-equivariant approach by leveraging an icosahedral group to learn a group of rotation-equivariant descriptors for each point. Rotational invariance is achieved by max-pooling over the group, but this method struggles with

Read the complete text