Fast Convergence Rates for Subsampled Natural Gradient Algorithms on Quadratic Model Problems
Gil Goldshlager, Jiang Hu, Lin Lin · 2025
Read Fast Convergence Rates for Subsampled Natural Gradient Algorithms on Quadratic Model Problems by Gil Goldshlager, Jiang Hu, Lin Lin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-21022.txt.
Description
Gil Goldshlager, Jiang Hu, and Lin Lin rethink the theory of a subsampled training method in a 2025 paper. Natural gradient descent rescales steps by a curvature map, and sampling part of the data makes it cheap enough for fine science work. Past proofs split gradients and rescalers into two lone batches, which misses how real runs with tiny samples behave.
Reading the method as sketch and project changes the view. Under a volume based sampling proxy, the mean step matches a rescaled gradient step even when parts couple. That yields global settling promises for a single batch of any size and a plain rate formula in sketch terms. The frame also explains a popular momentum scheme as a sped up sketch method. The authors say the edge over plain descent is sharper use of fading spectrum tails.
Catalog record
| CREATOR | Gil Goldshlager, Jiang Hu, Lin Lin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-21022