Riemannian Stochastic Optimization for Sufficient Dimension Reduction
Thibault Pautrel, François Portier · 2026
Read Riemannian Stochastic Optimization for Sufficient Dimension Reduction by Thibault Pautrel, François Portier (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-00413.txt.
Introduction
Pautrel and Portier speed up a shrink trick for high-width fits by a curved-space climb. The trick finds a low-width space that keeps the mean of a reply in full. Old tools work in the wide space and pay a vast cost per round. That cost grows with the square of the draw count in some.
Their take shows a famed risk's best fits aim at the same space as a slope tool. The fit task is recast as a climb on a curved frame with a closed slope. Their tool, SMAVE, blends scant near-point search with that curved climb. Proof gives firm settle marks for a plain form of it. Tests tie or beat a kin tool at far less run cost.
Catalog record
| CREATOR | Thibault Pautrel, François Portier |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 109 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.00413
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Riemannian Stochastic Optimization for Sufficient Dimension Reduction Thibault Pautrel, François Portier Abstract Sufficient dimension reduction (SDR) makes high-dimensional regression tractable by projecting the covariates onto a low-dimensional subspace that preserves the conditional mean of the response. Existing gradient-based estimators either operate in the ambient space and suffer from the curse of dimensionality, or localize in the reduced space at a per-outer-iteration cost at least quadratic in the sample size. We show that minimizers of the population Minimum Average Variance Estimation (MAVE) risk approximate the same Grassmannian target as the Outer Product of Gradients (OPG), and recast the empirical criterion as a smooth maximization on the Stiefel manifold with closed-form Riemannian gradient. The resulting algorithm, SMAVE, combines sparse projected-space nearest-neighbor localization with Riemannian stochastic gradient ascent. A simplified version comes with almost-sure convergence and a non-asymptotic rate matching the standard non-convex stochastic first-order scaling. Empirically, SMAVE matches or improves on RMAVE’s synthetic subspace recovery at moderate-to-high ambient dimension, and on four real datasets it uniformly improves over OPG and is competitive with or outperforms RMAVE at orders of magnitude lower runtime. † † affiliationnotice: Equal contribution 1 Introduction Sufficient dimension reduction (SDR) seeks a low-dimensional linear projection \mathbf{B}_{\ast}^{\top}X of high-dimensional covariates X\in\mathbb{R}^{p} that preserves all information relevant for predicting a response Y . Unlike unsupervised techniques such as principal component analysis, SDR exploits the supervised structure: the projection captures label-relevant variation rather than overall data variance. When such a projection exists with d\ll p , the intractable problem of estimating the regression function \mathbb{E}[Y\mid X] in \mathbb{R}^{p} reduces to the tractable problem of estimating \mathbb{E}[Y\mid\mathbf{B}_{\ast}^{\top}X] in \mathbb{R}^{d} . Beyond computational savings, this projection provides an interpretable, low-dimensional representation of the covariates for the supervised learning problem. This goal places SDR within a broader family of supervised representation learning methods. Metric learning provides a concrete example: learning a Mahalanobis distance d_{M}(x,x^{\prime})=\|x-x^{\prime}\|_{M} adapted to prediction tasks ( 21 ; 26 ) reduces to learning a linear projection when the metric matrix M has low rank. If \mathrm{rank}(M)=d\ll p , then M=\mathbf{B}\mathbf{B}^{\top} for some \mathbf{B}\in\mathbb{R}^{p\times d} , and d_{M}(x,x^{\prime})=\|\mathbf{B}^{\top}(x-x^{\prime})\|_{2} , which is precisely the Euclidean distance in the projected space that SDR methods seek to identify. Two broad families of methods have emerged for estimating the subspace spanned by \mathbf{B}_{\ast} from n independent samples, each navigating a fundamental tension between statistical flexibility and computational cost. Inverse regression approaches (including Sliced Inverse Regression ( 28 , SIR;) , Sliced Average Variance Estimation ( 13 , SAVE;) , and Directional Regression ( 27 , DR;) ) exploit moment conditions on \mathbb{E}[X\mid Y] to achieve \sqrt{n} -consistency with low computational cost, but require linearity or constant variance conditions that often fail in practice. Gradient-based methods offer greater flexibility by estimating the subspace directly from local derivative information, avoiding these distributional assumptions. The Average Derivative Estimator ( 22 ) and Outer Product of Gradients ( 36 , OPG;) construct subspace estimates from pointwise gradient estimates, while Minimum Average Variance Estimation ( 36 , MAVE;) formulates the problem as local polynomial regression. However, this flexibility comes at a computational price: the kernel bandwidth scale causes neighborhoods to become exponentially sparse when p is large. Refined variants like RMAVE ( 36 ) and structure-adaptive OPG ( 24 ; 15 ) address this by localizing in the projected space \mathbb{R}^{d} , but still incur a cost at least quadratic in n per refinement step from dense pairwise kernel weights. A separate line of work, initiated by 19 , characterizes dimension reduction subspaces through conditional independence in reproducing kernel Hilbert spaces ( 35 ; 32 ) , offering a distinct perspective that we do not pursue here. Since the target is a subspace rather than a specific matrix, the natural parameter space is the Grassmann manifold \mathrm{Gr}(p,d) of d -dimensional linear subspaces of \mathbb{R}^{p} . For computation, we represent each subspace by a matrix with orthonormal columns, optimizing over the Stiefel manifold \mathrm{St}(p,d)=\{\mathbf{B}\in\mathbb{R}^{p\times d}:\mathbf{B}^{\top}\mathbf{B}=\mathbf{I}_{d}\} . Riemannian optimization on these matrix manifolds is well-established in numerical linear algebra ( 17 ; 1 ) and increasingly adopted in machine learning, from decentralized training with orthogonality constraints ( 11 ) to parameter-efficient fine-tuning of large language models ( 31 ) . We bring these tools to sufficient dimension reduction, demonstrating that scalable Riemannian stochastic gradient methods can unlock both statistical and computational gains for foundational problems in nonparametric statistics. We propose SMAVE (Stochastic MAVE), a scalable algorithm that combines Riemannian optimization with adaptive localization in the projected space to achieve competitive or sharper subspace recovery than RMAVE at a fraction of the computational cost. Contributions. 1. Theoretical characterization. We establish that MAVE minimizers approximate the span of the regression function’s gradients (the same Grassmannian target as OPG) but recovered through projected-space local regression rather than ambient-space gradient aggregation (Propositions 2.4 – 2.5 ). This unifies two major gradient-based SDR families under a common target, and reveals that the MAVE objective is intrinsically a function on the Grassmannian, motivating a Riemannian treatment. 2. Riemannian reformulation. Building on the above, we reformulate MAVE as maximizing a smooth, \mathcal{O}(d) -invariant objective on \mathrm{St}(p,d) (Proposition 2.7 ) and derive its Riemannian gradient in closed form (Proposition 4.1 ). 3. Scalable algorithm with convergence guarantees. SMAVE combines mini-batch Riemannian stochastic gradient steps with sparse k -NN weights in the projected space, periodically refreshed as the subspace estimate evolves. This reduces per-iteration cost from the quadratic-in- n scaling of RMAVE’s outer iterations to a linear-in- n scaling of SMAVE. For a simplified version, we prove almost-sure convergence to a critical point (Proposition 4.3 ) and an O(1/\sqrt{mT}) non-asymptotic rate (Theorem 4.4 ), matching the optimal rate for non-convex stochastic first-order methods. 4. Empirical validation. On synthetic benchmarks, SMAVE reduces subspace estimation error over RMAVE while running one to two orders of magnitude faster, with the gap widening as the ambient dimension grows. On four real regression datasets spanning p\in[11,128] , SMAVE uniformly outperforms OPG, is broadly competitive with RMAVE at low and moderate p at a fraction of its runtime, and significantly outperforms RMAVE at p=128 . Organization. Section 2 formalizes the SDR problem and introduces our local approximation framework. Section 3 reviews related work, including gradient-based SDR methods and Riemannian optimization. Section 4 presents the SMAVE algorithm and establishes its convergence guarantees. Section 5 reports experiments on synthetic and real data. Proofs and additional results appear in the appendix. 2 From Sufficient Dimension Reduction to Local Regression 2.1 Sufficient Dimension Reduction Let (Y,X) be a random vector with Y\in\mathbb{R} and X\in\mathbb{R}^{p} , and let g(x):=\mathbb{E}[Y\mid X=x] denote the regression function. When p is large, estimating g directly suffers from the curse of dimensionality. The goal of sufficient dimension reduction is to find a matrix \mathbf{B}_{\ast}\in\mathbb{R}^{p\times d} with d\ll p such that \displaystyle g(X)=\mathbb{E}[Y\mid\mathbf{B}_{\ast}^{\top}X]. (1) When such a projection exists, the intractable problem of estimating g in \mathbb{R}^{p} reduces to estimation in the low-dimensional space \mathbb{R}^{d} . Remark 2.1 (Central mean subspace) . Condition ( 1 ) defines the central mean subspace , weaker than the classical central subspace requiring Y\perp X\mid\mathbf{B}_{\ast}^{\top}X . The central mean subspace suffices for regression and is the natural target of gradient-based methods ( 36 ) . Condition ( 1 ) exhibits a key invariance: it