Robust Point Cloud Reinforcement Learning via PCA-Based Canonicalization
Michael Bezick, Vittorio Giammarino, Ahmed H. Qureshi · 2025
Read Robust Point Cloud Reinforcement Learning via PCA-Based Canonicalization by Michael Bezick, Vittorio Giammarino, Ahmed H. Qureshi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20974.txt.
Introduction
Michael Bezick leads two co-authors on making reinforcement learning from point clouds robust to camera pose changes. Vision based control breaks under shifts in light, color, and viewpoint, and point clouds avoid some of that fragility yet still fail when the camera moves.
Their PPC framework canonicalizes each cloud, mapping any rigid rotation or shift to a unique standard pose through principal component alignment tailored for control. Policies then see a stable view of the scene geometry across viewpoints. The goal is robotic control that survives the messy camera placements of real deployments rather than only lab mounts.
Catalog record
| CREATOR | Michael Bezick, Vittorio Giammarino, Ahmed H. Qureshi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-20974
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Robust Point Cloud Reinforcement Learning via PCA-Based Canonicalization Michael Bezick, Vittorio Giammarino, Ahmed H. Qureshi Abstract Reinforcement Learning (RL) from raw visual input has achieved impressive successes in recent years, yet it remains fragile to out-of-distribution variations such as changes in lighting, color, and viewpoint. Point Cloud Reinforcement Learning (PC-RL) offers a promising alternative by mitigating appearance-based brittleness, but its sensitivity to camera pose mismatches continues to undermine reliability in realistic settings. To address this challenge, we propose PCA Point Cloud (PPC), a canonicalization framework specifically tailored for downstream robotic control. PPC maps point clouds under arbitrary rigid-body transformations to a unique canonical pose, aligning observations to a consistent frame, thereby substantially decreasing viewpoint-induced inconsistencies. In our experiments, we show that PPC improves robustness to unseen camera poses across challenging robotic tasks, providing a principled alternative to domain randomization. I Introduction Fig. 1 : Illustration of the proposed PCA Point Cloud (PPC) pipeline. Given an input point cloud, PPC first applies voxel downsampling to regularize point distribution, followed by Farthest Point Sampling (FPS) to select a fixed number of representative points, ensuring consistent shape coverage and enabling uniform batch processing. A PCA-based canonicalization is then performed, producing four candidate eigenvector bases, from which a geometry-driven disambiguation step deterministically selects one. The point cloud is subsequently re-expressed in this basis, yielding a unique disambiguated canonical pose. As a result, regardless of the initial rotation or translation of the input, the final canonical representation remains consistent. Reinforcement Learning (RL) from pixels, often referred to as Visual RL, aims to learn policies end-to-end by mapping high-dimensional image observations directly to actions. This paradigm has recently attracted considerable attention in robotics as it allows agents to acquire complex behaviors from sensory streams such as camera images or video sequences, removing the need for hand-crafted perception pipelines [ 1 , 2 , 3 , 4 ] . Yet the same reliance on raw visual input also exposes a critical weakness: pixel-based policies are notoriously brittle to distribution shifts, where even small changes in lighting, background, or camera pose can lead to severe performance degradation [ 5 , 6 , 7 ] . As a result, despite strong in-distribution performance, their lack of robustness to visual variations remains a major obstacle to real-world deployment. To overcome these limitations, recent work has turned to Point Cloud Reinforcement Learning (PC-RL) [ 8 ] , which defines policies directly on 3D point cloud observations. By leveraging geometric representations, PC-RL reduces sensitivity to nuisance factors such as lighting, background, and field of view [ 9 ] , and has proven particularly effective for tasks that require fine-grained reasoning about robot-object relationships [ 10 ] . Despite these advantages, point clouds are typically expressed in a camera’s local coordinate frame, making them inherently sensitive to sensor pose. Even modest translations or rotations can drastically change the scene representation, leading to instability and degraded performance at deployment. Consequently, while PC-RL alleviates appearance-related brittleness, it continues to suffer from viewpoint sensitivity. A common strategy to address this problem is given by domain randomization, where training environments are deliberately perturbed to encourage robustness. In the context of PC-RL, this typically involves applying random rotations and translations to point clouds so that policies are trained across a wide range of poses. By aiming to generalize across a vast distribution of perturbations, domain randomization often leads to substantial sample inefficiency. In addition, its theoretical guarantees are weak and rely on assumptions that are rarely satisfied in practice [ 11 , 12 ] . In this work, we present a principled alternative to domain randomization that tackles PC-RL’s viewpoint sensitivity both effectively and efficiently. Drawing on advances in rotation-invariant point cloud analysis [ 13 , 14 ] , we introduce PCA Point Cloud (PPC), a canonicalization technique based on Principal Component Analysis (PCA) that standardizes a point cloud’s reference frame, producing one consistent representation. Specifically, PPC aligns each point cloud with its principal axes and resolves PCA’s inherent sign ambiguity through a novel geometry-driven disambiguation step, ensuring that every input is mapped to a unique canonical pose. This guarantees invariance to both translation and rotation, effectively neutralizing PC-RL’s sensitivity to coordinate frames. We show that this simple yet rigorous procedure substantially improves the robustness of PointPatch RL (PPRL) [ 8 ] , enabling it to maintain strong performance under viewpoint shifts where the vanilla algorithm frequently collapses. We summarize our main contributions as follows: • We propose PCA Point Cloud (PPC) , a novel canonicalization method that maps each point cloud to a unique pose, guaranteeing invariance to rigid transformations. • We integrate PPC into PPRL [ 8 ] and demonstrate, through extensive experiments, substantial gains in robustness to unseen camera viewpoints across diverse robotic tasks. • We conduct additional tests on real-world point clouds using PPC to demonstrate how point cloud pose invariance effectively approximates camera pose invariance. II Related Work Point Cloud Representation Learning A central challenge in enabling end-to-end control from high-dimensional perceptual inputs is the design of robust representation learning pipelines that compress raw data into compact but informative embeddings. Such representations have underpinned the success of both model-free [ 1 , 15 ] and model-based [ 16 ] Visual RL. In the context of point clouds, early work focused on learning directly from unordered sets. PointNet [ 17 ] introduced permutation invariance via shared per-point MLPs and symmetric pooling, but struggled to capture local geometric structure. PointNet++ [ 18 ] improved upon this via hierarchical grouping and Farthest Point Sampling (FPS) to model local neighborhoods, though it remained sensitive to sampling density, scale, and rotations. Building on these foundations, PointTransformer [ 19 ] introduced vector attention mechanisms to capture interactions within local neighborhoods, while Point-MAE [ 20 ] adapted masked autoencoding by reconstructing randomly masked patches with a Chamfer distance loss. Point-BERT [ 21 ] replaced random masking with block masking and discrete token prediction, and Point-GPT [ 22 ] further extended this direction by adopting autoregressive inference strategies. Despite these advances, none of these methods provide robustness to rigid transformations such as translations or rotations, which remains a central challenge for their application to real-world control. Point Cloud Reinforcement Learning Existing work in PC-RL can be broadly categorized into: (i) approaches leveraging pretrained encoders, (ii) comparative studies against Visual RL, and (iii) fully end-to-end PC-RL methods. Early efforts explored pretrained PointNet features for manipulation without full end-to-end optimization [ 23 ] , while others adopted 3D CNNs with distillation [ 24 ] or integrated pretrained point cloud encoders into imitation learning pipelines, where finetuning occurred during the imitation step [ 25 ] . Beyond encoder design, Liu et al. [ 26 ] showed that the choice of coordinate frame can significantly affect policy performance in PC-RL. Comparative evaluations further demonstrated that PC-RL policies can outperform Visual RL in mixed 2D/3D environments [ 10 ] , underscoring the promise of 3D representations for control. More recently, specialized methods such as DexPoint [ 27 ] have advanced PC-RL for manipulation, while PPRL [ 8 ] introduced a patch-based encoder with auxiliary reconstruction objectives, establishing the state-of-the-art in PC-RL. All these existing methods remain sensitive to viewpoint mismatch, a critical limitation that motivates this