FA-KPConv: Introducing Euclidean Symmetries to KPConv via Frame Averaging Ali Alawieh 1 2 and Alexandru P. Condurache 1 2 Abstract We present Frame-Averaging Kernel-Point Convolution (FA-KPConv), a neural network architecture built on top of the well-known KPConv, a widely adopted backbone for 3D point cloud analysis. Even though invariance and/or equivariance to Euclidean transformations are required for many common tasks, KPConv-based networks can only approximately achieve such properties when training on large datasets or with significant data augmentations. Using Frame Averaging, we allow to flexibly customize point cloud neural networks built with KPConv layers, by making them exactly invariant and/or equivariant to translations, rotations and/or reflections of the input point clouds. By simply wrapping around an existing KPConv-based network, FA-KPConv embeds geometrical prior knowledge into it while preserving the number of learnable parameters and not compromising any input information. We showcase the benefit of such an introduced bias for point cloud classification and point cloud registration, especially in challenging cases such as scarce training data or randomly rotated test data. We plan to open-source our implementation upon publication. I Introduction Point cloud analysis has attracted a significant amount of research in recent years, especially given the widespread availability of point-cloud data. Being inherently unstructured, irregular and unordered, it is challenging to design neural network (NN) architectures to handle point clouds, and several approaches appeared over time. One of the well-known point cloud NN architectures is KPConv [ 1 ] , which has proven itself as a simple yet effective backbone, widely adopted until today for various point cloud analysis tasks [ 2 , 3 , 4 , 5 , 6 ] . Inspired by image convolution, KPConv defines point convolution using kernel points in Euclidean space, then applies the weights of those kernel points to the input points close to them. Fig. 1: Diagrams illustrating how (a) the equivariant function \hat{f}(\textbf{X},\textbf{F}) and (b) the invariant function \bar{f}(\textbf{X},\textbf{F}) can be obtained from f(\textbf{X},\textbf{F}) via frame averaging, which is key for defining FA-KPConv. We use small circles to denote the action of a certain group element g , and denote |\mathcal{F}(\textbf{X})| by N . Most of the known point cloud network architectures [ 7 ] were designed to ensure permutation invariance or permutation equivariance of the input data, which is crucial to deal with the inherent lack of order in the set of points constituting the point cloud. Furthermore, many tasks and use cases involve learning functions that are desired to be invariant or equivariant to other kinds of symmetries like rotations and translations. Being able to ensure such invariance or equivariance by design whenever needed is helpful, not only to embed and guarantee the desired properties, but also to improve the training sample efficiency and reduce the need for huge datasets or numerous data augmentations, while at the same time improving the explainability of the approach. By integrating a recent elegant method called Frame Averaging [ 8 ] into KPConv, we enable making the popular KPConv backbone invariant or equivariant by design to Euclidean transformations of the input data. We do so without any overhead of additional trainable parameters, which results in better usage of the available model capacity of the network to learn other relevant cues from the training data. II Related Work Point cloud neural network architectures were pioneered by PointNet [ 9 ] , which was the first architecture to directly operate on point clouds, i.e. without having to create a voxelized or a multi-view version of them. Essentially, it does so by passing each point of the point cloud through a shared multi-layer perceptron (MLP), then applying a permutation-invariant operation like max-pooling on the outputs to create a global feature vector for the point cloud. To address the limitations of PointNet in capturing local structures, follow-up works tried to alleviate those limitations by recursively sampling, grouping neighborhoods and learning local features per group, thereby hierarchically going from local to global features. Works like PointNet++ [ 10 ] and PointNeXt [ 11 ] employ MLPs per local neighborhood. Others like KPConv [ 1 ] , SpiderCNN [ 12 ] , PointCNN [ 13 ] , PAConv [ 14 ] and PointConv [ 15 ] define various flavors of convolution operations, and apply them per local neighborhood. Further methods like DGCNN [ 16 ] , ECC [ 17 ] and AdaptConv [ 18 ] model the point cloud as a graph, then define and apply graph convolution operations per local neighborhood. A more recent line of research embraces the transformer architecture and adapts it for point clouds, thereby globally extracting features while inherently capturing local and non-local structures, with the help of attention mechanisms, at the expense though of quadratic computational complexity. PCT [ 19 ] , PT [ 20 ] , PTv2 [ 21 ] , PTv3 [ 22 ] , Point-BERT [ 23 ] , Point-MAE [ 24 ] and Point-M2AE [ 25 ] are example works along this line. Even though KPConv [ 1 ] might not be the best performing among the mentioned architectures, yet it stands out for its intuitive design, inspired by image convolution, and for being a simple yet effective backbone still widely adopted in many recent works, and for various tasks [ 2 , 3 , 4 , 5 , 6 ] . The main idea behind KPConv is to place kernel points in 3D space distributed around a center point, which is the point being convolved. Each kernel point is associated with a learnable weight matrix, and defines a region in space where it interacts with neighboring input points based on proximity. The kernel points are then placed around each query point at which the output is to be computed, and, using all input points lying in a radius neighborhood of the query point, the output at that point equals the sum of contributions over all pairs that can be formed using one input point from the neighborhood and one kernel point. The contribution of each such pair is the product of the weight matrix of the kernel point and the feature vector of the input point, weighted by the proximity between the kernel and the input point. In- or equivariance to Euclidean transformations can either be approximately learned during training of a certain NN (e.g. via data augmentation), or introduced and enforced by design to its architecture. What approximate methods are concerned, there are several promising approaches: 1. One line of research focuses particularly on invariance and tries to approximately achieve it by estimating a transformation to align each input point cloud to a canonical space before feeding it into the NN. PointNet [ 9 ] for example employs what they call a joint alignment network, T-Net for short, which is a mini-version of PointNet itself aimed at regressing the affine transformation that aligns each input point cloud to a canonical space, thus achieving approximate \mathrm{SE}(3) invariance, which is desired for classification tasks. T-Net was inspired by spatial transformer networks (STNs) [ 26 ] from the image domain, and was also adopted in several follow-up works after PointNet. It was shown however that T-Net based approaches usually require a significant amount of data augmentations to learn the approximate invariance [ 27 ] . 2. Another line of research focuses on equivariance for convolutional neural networks (CNNs), as in Group equivariant Convolutional Neural Networks (G-CNNs) [ 28 ] and Steerable CNNs [ 29 ] that can be applied on regular images, and in spherical CNNs [ 30 ] for spherical images. For point clouds, Equivariant Point Network (EPN) [ 31 ] combines group convolution with point convolution to achieve \mathrm{SE}(3) equivariance. Even though those methods define group convolutions using group theory, which mathematically guarantees exact rotation equivariance, they are in practice applied on a finite group of rotations such as icosahedral grids, not over the intractable infinite continuous group of rotations. That’s why they can still be considered as approximate methods. What exact methods are concerned, there are also several promising approaches: 1. A first line of research was pioneered by the seminal work entitled Tensor Field Networks (TFNs) [ 32 ] . By relying on irreducible representations of the rotation group, TFN designs convolution layers that are \mathrm{SE}(3) -equivariant by ensuring that both feature maps and learned filters transform consistently under \mathrm{SE}(3) transformations. Another known work in that direction is \mathrm{SE}(3) -Transformer [ 33 ] which extends TFN from convolution to transformer layers, with equivariant self-interaction and message passing / attention mechanisms. It is worth noting that this set of methods is mathematically quite involved and complex. 2. Another line of research achieves exact invariance by hand-crafting, from the input point clouds, values that are invariant by design. Example invariant values are distances, angles, dot and cross products between vectors defined from the points lying in a certain neighborhood, or from the normal vectors at those points, as done in [ 34 , 35 , 36 , 37 ] . Another way to define invariant values is by first constructing one or more rotation-equivariant local reference frame(s) (LRF) like the Darboux frame [ 38 ] , then projecting relative point coordinates to the LRF(s) to render them \mathrm{SE}(3) -invariant. 3. A third line of research achieves exact \mathrm{SE}(3) in- or equivariance by defining or modifying NN layers to make them as such. One prominent example is Equivariant Graph Neural Networks (EGNN) [ 39 ] , in which the message passing operations of the graph neural network are designed to update both node features and relative coordinates between the nodes in a way that preserves equivariance. Another seminal work is Vector Neurons (VN) [ 40 ] , which represents features as 3-dimensional vectors rather than scalars, and maintains those representations throughout the network by building equivariance into common NN layers (like linear, non-linear, pooling and normalization layers). 4. A unique approach enjoying the advantages and avoiding the disadvantages of the other approaches is called Frame Averaging [ 8 ] . It does not require designing any dedicated NN layers, and is applicable to any available point cloud NN via simple plug-and-play, without having to modify its layers, or to add any parameters to be learned to it. It is also exact, not approximate, since it is based on group theory, and avoids having to hand-craft input features and the potential information loss associated to that. The idea behind it is to simply replace group averaging, which is intractable for an infinite group like \mathrm{SE}(3) with frame averaging, where a frame is a carefully selected and input-dependent small subset of that group, on which averaging is shown to be exactly equivalent to averaging over the whole group. Further insights on methods for embedding geometrical prior knowledge into NNs are surveyed in [ 41 ] , a recent survey on that field. Our contribution in this work consists mainly of the following: • We create FA-KPConv by integrating F rame A veraging with the widely-used KPConv backbone, thereby enabling users to easily embed geometrical prior knowledge into that popular architecture. More specifically, we enable the introduction of exact invariance or equivariance to Euclidean transformations of the input point cloud. • We showcase the performance boost, the improved learning capacity and the sample efficiency benefits of FA-KPConv on point cloud classification and registration tasks, especially for geometrically transformed datasets and in the low-data regime. III Methodology Our approach integrates Frame Averaging [ 8 ] and KPConv [ 1 ] into FA-KPConv. Details of the approach are provided in the following sections. III-A Frame Averaging for Point Cloud Networks We start by recalling some basics related to equivariance, invariance and group theory [ 41 ] . A function f:X\rightarrow Y is equivariant with respect to a transformation group G if, for any input \textbf{X}\in X and transformation g\in G , transforming X by g then feeding g\cdot\textbf{X} to f gives the same result as feeding X to f then transforming f(\textbf{X}) by g^{\prime} , where g^{\prime} is the transformation corresponding to g in the output space Y , i.e.: f(g\cdot\textbf{X})=g^{\prime}\cdot f(\textbf{X})\qquad\forall\textbf{X}\in X,\;\forall g\in G (1) Invariance on the other hand is a special case of equivariance, where the transformation in output space is the identity, i.e. the output is unaffected by transformations g\in G : f(g\cdot\textbf{X})=f(\textbf{X})\qquad\forall\textbf{X}\in X,\;\forall g\in G (2) Moreover, an arbitrary function f:X\rightarrow Y can be made equivariant or invariant to a certain transformation group G by symmetrization , that is averaging over the group [ 41 ] , i.e. starting from f , we can obtain an equivariant function \hat{f} or an invariant function \bar{f} as such: \hat{f}(\textbf{X})=\frac{1}{|G|}\sum_{g\in G}{g\cdot f(g^{-1}\cdot\textbf{X})} (3) \bar{f}(\textbf{X})=\frac{1}{|G|}\sum_{g\in G}{f(g^{-1}\cdot\textbf{X})} (4) Symmetrization becomes intractable when the cardinality of G is large or infinite, as is the case for the continuous group of 3D translations or rotations. This intractability made many works resort to approximate in- or equivariance by averaging over a subset of the transformation group [ 28 , 29 , 31 ] . Frame Averaging [ 8 ] elegantly avoids both issues by averaging over a small input-dependent subset of transformations, designed and chosen in a way that guarantees exact equivalence between averaging over it and the intractable averaging over the whole group. That subset is called a frame \mathcal{F}(\textbf{X}) , and equations 3 & 4 become: \hat{f}(\textbf{X})=\frac{1}{|\mathcal{F}(\textbf{X})|}\sum_{g\in\mathcal{F}(\textbf{X})}{g\cdot f(g^{-1}\cdot\textbf{X})} (5) \bar{f}(\textbf{X})=\frac{1}{|\mathcal{F}(\textbf{X})|}\sum_{g\in\mathcal{F}(\textbf{X})}{f(g^{-1}\cdot\textbf{X})} (6) TABLE I: Frames for groups of Euclidean motions in \mathbb{R}^{d} . \mathrm{O}(d) , \mathrm{SO}(d) , \mathrm{E}(d) and \mathrm{SE}(d) denote respectively the orthogonal, special orthogonal, Euclidean and special Euclidean groups. Group Frame Name Elements Representation for \textbf{X}\in\mathbb{R}^{n\times d} \mathcal{F}(\textbf{X}) |\mathcal{F}(\textbf{X})| Translations \textbf{t}\in\mathrm{T}(d)=\mathbb{R}^{d} \textbf{X}+\textbf{1}\textbf{t}^{T} \{\textbf{c}\} 1 Rotations \textbf{R}\in\mathrm{SO}(d) \textbf{XR}^{T} \{\textbf{Q}\mid\textbf{C}=\textbf{Q}\Lambda\textbf{Q}^{T},|\textbf{Q}|=1\} 2^{d-1} Rotations & Reflections \textbf{R}\in\mathrm{O}(d) \textbf{XR}^{T} \{\textbf{Q}\mid\textbf{C}=\textbf{Q}\Lambda\textbf{Q}^{T}\} 2^{d} Translations & Rotations (\textbf{R},\textbf{t})\in\mathrm{SE}(d) \textbf{XR}^{T}+\textbf{1}\textbf{t}^{T} \{(\textbf{Q},\textbf{c})\mid\textbf{C}=\textbf{Q}\Lambda\textbf{Q}^{T},|\textbf{Q}|=1\} 2^{d-1} Translations & Rotations & Reflections (\textbf{R},\textbf{t})\in\mathrm{E}(d) \textbf{XR}^{T}+\textbf{1}\textbf{t}^{T} \{(\textbf{Q},\textbf{c})\mid\textbf{C}=\textbf{Q}\Lambda\textbf{Q}^{T}\} 2^{d} Such a frame might not exist though for all possible functions f:X\rightarrow Y and transformation groups G . Reference [ 8 ] proves the existence of such frames for any function that operates on point clouds (thus including point cloud NNs) and for groups of Euclidean motions in \mathbb{R}^{d} . Formally, for a point cloud \textbf{X}\in\mathbb{R}^{n\times d} , n being the number of points in the point cloud, and using 1 to denote a vector of ones \in\mathbb{R}^{n} , we first compute the point cloud centroid c and the covariance matrix C as such: \textbf{c}=\frac{1}{n}\textbf{X}^{T}\textbf{1} (7) \textbf{C}=(\textbf{X}-\textbf{1}\textbf{c}^{T})^{T}(\textbf{X}-\textbf{1}\textbf{c}^{T}) (8) We then obtain the matrix Q by normalized Eigendecomposition of C : \textbf{C}=\textbf{Q}\Lambda\textbf{Q}^{T} (9) Note that the columns of Q consist of unit-length eigenvectors, and are unique up to a sign change. Thus, we have 2^{d} possible solutions for Q in equation ( 9 ). Using c and Q , the frames for the groups of Euclidean motions in \mathbb{R}^{d} are listed in Table I . Further details and proofs can be found in [ 8 ] . III-B KPConv Starting from a point cloud whose coordinates are represented as \textbf{X}\in\mathbb{R}^{n\times d} and features as \textbf{F}^{in}\in\mathbb{R}^{n\times c_{in}} , KPConv [ 1 ] computes the output feature vector \textbf{f}^{out}\in\mathbb{R}^{c_{out}} at a query point \textbf{q}\in\mathbb{R}^{d} using point convolution by a kernel h as such: \textbf{f}^{out}=\sum_{i\in\mathcal{N}_{\textbf{q}}}{h(\textbf{X}_{i}-\textbf{q})\textbf{F}_{i}^{in}} (10) where \mathcal{N}_{\textbf{q}} is the radius neighborhood of q from X for a certain radius r , i.e. \mathcal{N}_{\textbf{q}}=\{j\in\{0,1,\dots,n-1\}\mid\|\textbf{X}_{j}-\textbf{q}\|\leq r\} . To define h , we first define \{(\tilde{\textbf{x}}_{k},\textbf{W}_{k})\in\mathbb{R}^{d}\times\mathbb{R}^{c_{in}\times c_{out}}\mid k