Full-Batch Gradient Descent Outperforms One-Pass SGD: Sample Complexity Separation in Single-Index Learning

Filip KovačevićHong Chang Ji, Denny Wu, Mahdi Soltanolkotabi, Marco Mondelli · 2026

Read Full-Batch Gradient Descent Outperforms One-Pass SGD: Sample Complexity Separation in Single-Index Learning by Filip KovačevićHong Chang Ji, Denny Wu, Mahdi Soltanolkotabi, Marco Mondelli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02431.txt.

Description

Filip Kovačević and 4 co-authors prove that reusing training data can beat one-pass learning in a nonlinear setting, in a 2026 arXiv paper. Folklore says multiple passes improve statistical efficiency, but the benefit was murky outside linear regression. Their test case is a single-index model, where the outcome depends on one hidden direction, with a quadratic activation.

One-pass stochastic gradient descent needs slightly more than dimension-many samples for weak recovery. Full-batch gradient descent on a correlation loss keeps that extra factor, the authors show. But truncating the activation changes the landscape: full-batch descent then succeeds with roughly dimension-many samples. A trajectory analysis on the squared loss shows strong, exact recovery with enough samples and logarithmically many steps. The separation is a clean theoretical case where data reuse provably pays.

Catalog record

CREATORFilip KovačevićHong Chang Ji, Denny Wu, Mahdi Soltanolkotabi, Marco Mondelli
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE226 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-02431

Browse the full library · About the Tower of Babel