Dual Space Preconditioning for Gradient Descent in the Overparameterized Regime

Reza Ghane, Danil Akhtiamov, Babak Hassibi · 2026

Read Dual Space Preconditioning for Gradient Descent in the Overparameterized Regime by Reza Ghane, Danil Akhtiamov, Babak Hassibi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-10485.txt.

Description

Reza Ghane, Danil Akhtiamov, and Babak Hassibi study dual space preconditioned gradient descent, a family that includes normalized gradient descent and gradient clipping, in the overparameterized regime. For a linear model with convex loss, they prove full-batch iterates converge exponentially fast to a point that fits the data exactly. That limit point depends on step size for general preconditioners, blurring any neat implicit bias story. They give an approximate bias bound there, and exact convergence to the plain descent limit for isotropic preconditioners, in batch and stochastic forms.

Catalog record

CREATORReza Ghane, Danil Akhtiamov, Babak Hassibi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50603
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.10485

Browse the full library · About the Tower of Babel