Training Without Orthogonalization, Inference With SVD: A Gradient Analysis of Rotation Representations

Chris Choy · 2026

Read Training Without Orthogonalization, Inference With SVD: A Gradient Analysis of Rotation Representations by Chris Choy (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05414.txt.

Introduction

Choy explains why dropping a step called orthogonalization from training, and saving it for inference, improves rotation estimates in deep learning. His study of SVD gradients for 3-by-3 matrices finds a rank-3 backward pass whose distortion peaks when predictions sit far from true rotations. He also shows Gram-Schmidt gradients skew unevenly across parameters, backing direct 9D regression plus SVD only at inference.

Catalog record

CREATORChris Choy
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76035
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05414

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Training Without Orthogonalization, Inference With SVD: A Gradient Analysis of Rotation Representations Chris Choy Abstract Recent work has shown that removing orthogonalization during training and applying it only at inference improves rotation estimation in deep learning, with empirical evidence favoring 9D representations with SVD projection ( Gu et al. 2024 ) . However, the theoretical understanding of why SVD orthogonalization specifically harms training, and why it should be preferred over Gram-Schmidt at inference, remains incomplete. We provide a detailed gradient analysis of SVD orthogonalization specialized to 3 × 3 3\times 3 matrices and SO ⁡ ( 3 ) \mathrm{SO}(3) projection. Our central result derives the exact spectrum of the SVD backward pass Jacobian: it has rank 3 3 (matching the dimension of SO ⁡ ( 3 ) \mathrm{SO}(3) ) with nonzero singular values 2 / ( s i + s j ) 2/(s_{i}+s_{j}) and condition number κ = ( s 1 + s 2 ) / ( s 2 + s 3 ) \kappa=(s_{1}+s_{2})/(s_{2}+s_{3}) , creating quantifiable gradient distortion that is most severe when the predicted matrix is far from SO ⁡ ( 3 ) \mathrm{SO}(3) (e.g., early in training when s 3 ≈ 0 s_{3}\approx 0 ). We further show that even stabilized SVD gradients ( Wang et al. 2022 ) introduce gradient direction error, whereas removing SVD from the training loop avoids this tradeoff entirely. We also prove that the 6D Gram-Schmidt Jacobian has an asymmetric spectrum: its parameters receive unequal gradient signal, explaining why 9D parameterization is preferable. Together, these results provide the theoretical foundation for training with direct 9D regression and applying SVD projection only at inference. 1 Introduction Representing 3D rotations for deep learning is a fundamental problem in computer vision and robotics. A neural network must output a rotation R\in\mathrm{SO}(3) , but the rotation group is a 3-dimensional manifold embedded in \mathbb{R}^{3\times 3} with the orthogonality constraint R^{\top}R=\mathbf{I},\det(R)=1 . This has led to diverse representations: axis-angle, quaternions, 6D ( Zhou et al. 2019 ) , and 9D with SVD projection ( Levinson et al. 2020 ) . Two recent lines of work have reached different conclusions about when orthogonalization should be applied. Levinson et al. 2020 showed that SVD orthogonalization is the maximum likelihood estimator under isotropic Gaussian noise and, to first order, produces half the expected reconstruction error of Gram-Schmidt, advocating for SVD as a differentiable layer during both training and inference. We call this SVD-Train (following the terminology of Levinson et al. 2020 ). We use SVD-Inference to denote the alternative where SVD is applied only at inference, with the training loss computed on the raw (unorthogonalized) matrix. Gu et al. 2024 showed that any orthogonalization during training introduces gradient pathologies (ambiguous updates, exploding gradients, suboptimal convergence) and proposed learning “pseudo rotation matrices” (PRoM), which is an instance of SVD-Inference: direct 9D regression during training with SVD projection applied only at test time. However, key questions remain unanswered. The PRoM analysis treats all orthogonalizations uniformly through a general non-injectivity argument, without dissecting the specific gradient failure modes of SVD versus Gram-Schmidt. While their ablation study shows that SVD at inference outperforms Gram-Schmidt (54.8 vs. 55.6 PA-MPJPE), no theoretical justification is provided for this choice. Moreover, no prior work has formally analyzed why 9D regression is preferable to 6D regression in the unorthogonalized training regime. We address these questions through a detailed Jacobian analysis of the SVD and Gram-Schmidt orthogonalization mappings. Our primary contribution is a detailed analysis of SVD gradient pathology specialized to 3\times 3 matrices and \mathrm{SO}(3) projection: • We derive the exact spectrum of the SVD Jacobian: it has rank 3 with nonzero singular values 2/(s_{i}+s_{j}) for each pair i<j , giving spectral norm 2/(s_{2}+s_{3})=O(1/\delta) and condition number \kappa=(s_{1}+s_{2})/(s_{2}+s_{3}) ( Theorem 1 ). This significantly extends PRoM’s qualitative observation about the K matrix by providing the exact spectral characterization. • We show that even state-of-the-art stabilization ( Wang et al. 2022 ) cannot eliminate this pathology without introducing gradient direction error, making avoidance strictly preferable to mitigation ( Remark 3 ). • We quantify the gradient information loss : SVD backpropagation retains only 1/3 of gradient energy, discarding the 6-dimensional normal component that encodes distance from \mathrm{SO}(3) ( Proposition 1 ). We also prove that Gram-Schmidt’s Jacobian has an asymmetric spectrum ( Theorem 2 ), explaining why 9D is preferable to 6D. Combined with SVD’s optimality as an inference-time projector ( 3\times error reduction, Corollary 2 ), these results explain why 9D + SVD-inference works. Figure 1: Gradient of \mathcal{L} w.r.t. M_{11} (or t^{\prime}_{11} ) vs. the error M_{11}-R^{*}_{11} , with \sigma=0.5 Gaussian noise ( 10{,}000 samples). Left: Direct 9D gradients lie on the diagonal (each element depends only on its own error). Center: SVD-Train gradients scatter across all quadrants; color encodes the singular value gap \delta=s_{2}+s_{3} (small \delta in red = most erratic). Right: GS-Train (6D) also produces ambiguous gradients from cross-column coupling. 2 Related Work Rotation representations. Classical parameterizations (Euler angles, axis-angle, quaternions) are all discontinuous mappings from \mathrm{SO}(3) to \mathbb{R}^{d} for d\leq 4 , a topological necessity proved by Stuelpnagel 1964 . Zhou et al. 2019 proposed the first continuous representation for neural networks: a 6D parameterization using two columns of the rotation matrix, recovered via Gram-Schmidt orthogonalization. Levinson et al. 2020 subsequently advocated for a 9D representation with SVD orthogonalization, showing it is the maximum likelihood estimator under Gaussian noise and produces half the expected error of Gram-Schmidt. Alternative approaches include spherical regression on n -spheres ( Liao et al. 2019 ) , smooth quaternion representations ( Peretroukhin et al. 2020 ) , mixed classification-regression frameworks ( Mahendran et al. 2018 ) , probabilistic models using von Mises ( Prokudin et al. 2018 ) or matrix Fisher distributions ( Gilitschenski et al. 2020 ) , and direct manifold regression ( Brégier 2021 ) . Both Gilitschenski et al. 2020 and Brégier 2021 use unconstrained parameterizations, avoiding orthogonalization constraints during training. Geist et al. 2024 provide a comprehensive survey of rotation representations, empirically observing that SVD gradient ratios between columns stay closer to 1 than GS (their Fig. 7), and framing SVD as an “ensemble” where all columns contribute equally. Our work provides the rigorous mathematical foundations for these empirical observations. We note that Geist et al. 2024 report that directly predicting rotation matrix entries is worse than using SVD/GS layers; however, their comparison uses orthogonalization-based losses during training, not the MSE-on-raw-matrix approach of PRoM ( Gu et al. 2024 ) , which is the key distinction. Orthogonalization during training. Gu et al. 2024 showed that incorporating any orthogonalization (Gram-Schmidt or SVD) during training introduces gradient pathologies: ambiguous updates, exploding gradients, and provably suboptimal convergence. They proposed Pseudo Rotation Matrices (PRoM), which remove orthogonalization during training and apply it only at inference, achieving state-of-the-art results on human pose estimation benchmarks. PRoM’s theoretical contributions are two-fold: (i) a convergence bound showing that the loss gap L(\hat{R}_{i})-L(R^{*}) is controlled by \psi(B_{i})=1/\sqrt{\lambda_{\min}(B_{i}B_{i}^{\top})} , and (ii) a proof that \psi(B_{1})=\infty for any non-locally-injective orthogonalization g , while \psi(B_{0})=1 when g is removed. These results are general: they treat Gram-Schmidt and SVD identically as instances of non-injective g , without deriving explicit bounds on SVD gradient magnitude, conditioning, or directional distortion. PRoM

Read the complete text