Factor Augmented High-Dimensional SGD

Shubo Li, Yuefeng Han, Xiufan Yu · 2026

Read Factor Augmented High-Dimensional SGD by Shubo Li, Yuefeng Han, Xiufan Yu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19291.txt.

Introduction

Shubo Li, Yuefeng Han, and Xiufan Yu introduce factor-augmented stochastic gradient descent for high-dimensional learning. Standard two-stage methods first learn representations offline and store full data, which scales poorly. Their method instead works purely on streaming data, weaving latent factor structure directly into the optimization steps. They also supply the first theory that folds factor-estimation error into stochastic gradient analysis, proving moment convergence under shrinking step sizes and mini-batch updates. The result gives a firmer footing for reliable large-scale learning.

Catalog record

CREATORShubo Li, Yuefeng Han, Xiufan Yu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58686
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19291

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Factor Augmented High-Dimensional SGD Shubo Li, Yuefeng Han, Xiufan Yu Abstract Stochastic gradient descent (SGD) is a fundamental optimization algorithm widely used in modern machine learning. In this paper, we propose Factor-Augmented SGD (FSGD), a new optimization method that leverages latent factor representations in high-dimensional learning tasks. Unlike standard two-stage dimension reduction approaches that rely on offline representation learning and full data storage, a key novelty of FSGD is that it operates purely on streaming data, making it scalable to large-scale and high-dimensional problems. Furthermore, we establish the first theoretical framework that explicitly incorporates latent factor estimation error into the analysis of SGD, and provide moment convergence in ℓ s \ell^{s} norm under decaying step sizes and mini-batch updates. Our results provide a new foundation for employing SGD reliably and scalably in high-dimensional machine learning systems. 1 Introduction Stochastic gradient descent (SGD) has been a cornerstone of machine learning since the pioneering work of 39 . Beyond its algorithmic simplicity and scalability, SGD has also become a central object of theoretical study, with refined analyses linking its dynamics to implicit regularization, generalization performance, and algorithmic stability. For decades, theoretical analyses of SGD have largely resided within the realm of classical stochastic approximation ( 38 ; 26 ; 7 ) , where the data dimension is considered fixed while the sample size tends to infinity. While this regime has yielded foundational insights, it no longer fully reflects the characteristics of modern learning systems. Contemporary applications often operate in regimes where data dimension, sample size, and model complexity grow together, calling for new theoretical tools and perspectives that go beyond traditional asymptotic analyses. In this study, we focus on the learning tasks involving high-dimensional predictors. When SGD is applied directly to such data, the dimensionality of the feature space propagates into the optimization process, resulting in a high-dimensional (HD) parameter space. Algorithmically, one trending strategy is to approximate the gradient updates using a low-rank representation to reduce memory costs and accelerate computation ( 47 ; 45 ; 25 ; 24 ; 53 ) . Theoretically, despite the vast literature on SGD, convergence guarantees of HD-SGD remain limited ( 16 ; 28 ) . Despite recent progress, the behavior of SGD in HD regimes remains only partially understood, and its direct application to high-dimensional datasets can be empirically fragile. Constructing models directly in high-dimensional feature spaces therefore leads to excessive parameter complexity and exacerbates the curse of dimensionality ( 2 ; 41 ) . Moreover, models defined on high-dimensional feature spaces often involve a large number of parameters. Applying SGD in such high-dimensional parameter regimes is computationally costly and is known to exhibit unstable or oscillatory behavior around flat minima ( 20 ; 48 ; 12 ) . To address the challenges induced by HD predictors, instead of directly optimizing HD-SGD in the high-dimensional feature space, we propose a new strategy, named Factor-Augmented Stochastic Gradient Descent (FSGD) , that augments SGD with a factor model. The key insight is that the effective information content of HD predictors often lies in a much lower-dimensional latent space, which can be captured by the “factors”. Ignoring this structure leads to inefficient updates that waste computation on noisy directions, slow convergence along dominant subspaces, and obscure the true geometry of the optimization landscape. By modeling predictors dynamically through a factor decomposition, we can extract useful representations from predictors and design SGD updates that adapt to the intrinsic dimensionality of the problem. Integrating factor models into SGD is far from a straightforward adaptation, but involves several non-trivial challenges to be carefully addressed. First, existing factor-augmented learning approaches ( 13 ; 54 ; 55 ; 31 ; 32 ) in other applications typically adopt a two-stage paradigm, where a low-dimensional representation is first learned using an offline procedure, and then treated as fixed during downstream learning tasks. Such two-stage approaches rely on full access to the dataset and require storing or repeatedly scanning all samples, which is inefficient for iterative SGD updates and may even be infeasible in large-scale learning settings where data arrive sequentially and memory resources are limited. To tackle this challenge, we propose an online factor-learning procedure designed for a fully streaming setting and seamlessly integrate it with SGD updates, enabling memory-efficient and scalable learning. Second, a key advantage of factor models is that they explicitly allow for the presence of idiosyncratic errors, capturing variation in predictors that is not explained by a small number of common factors. Most existing works in representation learning literature focus exclusively on learning low-dimensional embeddings and implicitly absorb idiosyncratic components into the representation itself, ignoring both the error structure and the associated estimation uncertainty. There is limited understanding of how idiosyncratic noise and representation estimation error interact with stochastic optimization procedures, particularly when SGD is employed. Our work bridges this gap by rigorously quantifying how representation estimation error and idiosyncratic noise propagate into the convergence rate. These limitations raise a fundamental question: Can the latent factor structure of high-dimensional predictors be explicitly utilized to improve the efficiency and theoretical understanding of SGD in high dimensions? In this work, we provide a positive answer to the above question. We summarize our main contributions as follows. 1. Factor-Augmented SGD (FSGD). We propose a new optimization method that utilizes the latent structure of high-dimensional predictors. Specifically, our method learns a low-dimensional latent representation via online PCA and simultaneously updates the model parameters using SGD on the induced factor space. While our theory is developed for strongly convex population objectives, the neural network experiments illustrate the empirical robustness of the FSGD principle beyond the convex setting. 2. Fully streaming unsupervised factor learning coupled with SGD. We develop a fully streaming learning framework in which both unsupervised latent factors and model parameters are updated using SGD-type procedures. Unlike classical two-stage representation learning approaches that require an offline PCA step with full data access, FSGD operates on streaming data and does not store the entire dataset, crucial for practical large-scale learning scenarios. As a result, factor estimates can be continuously refined during model training rather than being fixed a priori, causing the population minimizer to shift over time. In Proposition 3.5 , we establish a novel bound for minimizer shift to address the nontrivial statistical dependence between factor estimation and gradient updates. 3. First theoretical guarantee for SGD with latent factor estimation. To the best of our knowledge, Theorem 3.6 provides the first q -th moment convergence analysis of SGD that accounts for both latent factor estimation error and idiosyncratic noise in high-dimensional settings. Unlike existing approaches that treat the learned representation as fixed or known and ignore the idiosyncratic component in the factor structure, we rigorously quantify how the representation estimation error and idiosyncratic noise propagate into the convergence rate of SGD. 1.1 Related Works HD-SGD. While the theory of SGD has been extensively studied in traditional low-dimensional settings ( 7 ; 16 ) , theoretical understanding of HD-SGD remains underdeveloped, with only a handful of recent works. 35 ; 36 ; 37 study HD-SGD under quadratic loss functions. 28 studies HD-SGD and its Ruppert-Polyak averaged variant, and provides guarantees on moment convergence and high-probability concentration bounds in HD regimes with constant learning rates. Our

Read the complete text