Streaming Krylov-Accelerated Stochastic Gradient Descent
Stephen Thomas · 2025
Read Streaming Krylov-Accelerated Stochastic Gradient Descent by Stephen Thomas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07046.txt.
Description
Stephen Thomas presents a 2025 paper on speeding up the workhorse training method for machine learning when the problem is badly scaled. Stochastic gradient descent crawls on ill-conditioned problems, where some directions matter far more than others. His method, SKA-SGD, projects each noisy gradient step onto a small Krylov subspace, a smart low-dimensional arena where progress is faster.
Three tricks make it practical: projection coefficients from a single streaming solver pass, a Chebyshev basis for numerical stability, and code tuned for AMD graphics cards. Theory shows near machine-precision accuracy at lower cost than classical approaches. In tests the method beat both plain descent and the popular Adam optimizer, especially when condition numbers passed a thousand.
Catalog record
| CREATOR | Stephen Thomas |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 79 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-07046