Fast Convergence Rates for Subsampled Natural Gradient Algorithms on Quadratic Model Problems

Gil Goldshlager, Jiang Hu, Lin Lin · 2025

Read Fast Convergence Rates for Subsampled Natural Gradient Algorithms on Quadratic Model Problems by Gil Goldshlager, Jiang Hu, Lin Lin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-21022.txt.

Description

Gil Goldshlager, Jiang Hu, and Lin Lin rethink the theory of a subsampled training method in a 2025 paper. Natural gradient descent rescales steps by a curvature map, and sampling part of the data makes it cheap enough for fine science work. Past proofs split gradients and rescalers into two lone batches, which misses how real runs with tiny samples behave.

Reading the method as sketch and project changes the view. Under a volume based sampling proxy, the mean step matches a rescaled gradient step even when parts couple. That yields global settling promises for a single batch of any size and a plain rate formula in sketch terms. The frame also explains a popular momentum scheme as a sped up sketch method. The authors say the edge over plain descent is sharper use of fading spectrum tails.

Catalog record

CREATORGil Goldshlager, Jiang Hu, Lin Lin
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-21022

Browse the full library · About the Tower of Babel