Streaming Krylov-Accelerated Stochastic Gradient Descent

Stephen Thomas · 2025

Read Streaming Krylov-Accelerated Stochastic Gradient Descent by Stephen Thomas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07046.txt.

Description

Stephen Thomas presents a 2025 paper on speeding up the workhorse training method for machine learning when the problem is badly scaled. Stochastic gradient descent crawls on ill-conditioned problems, where some directions matter far more than others. His method, SKA-SGD, projects each noisy gradient step onto a small Krylov subspace, a smart low-dimensional arena where progress is faster.

Three tricks make it practical: projection coefficients from a single streaming solver pass, a Chebyshev basis for numerical stability, and code tuned for AMD graphics cards. Theory shows near machine-precision accuracy at lower cost than classical approaches. In tests the method beat both plain descent and the popular Adam optimizer, especially when condition numbers passed a thousand.

Catalog record

CREATORStephen Thomas
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-07046

Browse the full library · About the Tower of Babel