Full-Batch Gradient Descent Outperforms One-Pass SGD: Sample Complexity Separation in Single-Index Learning
Filip KovačevićHong Chang Ji, Denny Wu, Mahdi Soltanolkotabi, Marco Mondelli · 2026
Read Full-Batch Gradient Descent Outperforms One-Pass SGD: Sample Complexity Separation in Single-Index Learning by Filip KovačevićHong Chang Ji, Denny Wu, Mahdi Soltanolkotabi, Marco Mondelli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02431.txt.
Description
Filip Kovačević and 4 co-authors prove that reusing training data can beat one-pass learning in a nonlinear setting, in a 2026 arXiv paper. Folklore says multiple passes improve statistical efficiency, but the benefit was murky outside linear regression. Their test case is a single-index model, where the outcome depends on one hidden direction, with a quadratic activation.
One-pass stochastic gradient descent needs slightly more than dimension-many samples for weak recovery. Full-batch gradient descent on a correlation loss keeps that extra factor, the authors show. But truncating the activation changes the landscape: full-batch descent then succeeds with roughly dimension-many samples. A trajectory analysis on the squared loss shows strong, exact recovery with enough samples and logarithmically many steps. The separation is a clean theoretical case where data reuse provably pays.
Catalog record
| CREATOR | Filip KovačevićHong Chang Ji, Denny Wu, Mahdi Soltanolkotabi, Marco Mondelli |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 226 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02431