Does Weight Decay Enhance Training Stability?
Marius SaetherNTNU, MIT [email protected], Amir KolicMIT [email protected] Tomaso PoggioMIT [email protected] Pierfrancesco BeneventanoMIT [email protected] · 2026
Read Does Weight Decay Enhance Training Stability? by Marius SaetherNTNU, MIT [email protected], Amir KolicMIT [email protected] Tomaso PoggioMIT [email protected] Pierfrancesco BeneventanoMIT [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16622.txt.
Description
Saether, Kolic, Poggio and Beneventano test whether weight decay, the usual penalty on large weights, truly stabilizes deep-network training. They show it reliably slows progressive sharpening of the loss. Then they uncover an architecture split. It damps oscillations at the edge of stability in convolutional networks, while in multilayer perceptrons it drives sharpness to a threshold well below the classical boundary. A framework built on alignment between weights and the sharpness gradient explains the split.
Catalog record
| CREATOR | Marius SaetherNTNU, MIT [email protected], Amir KolicMIT [email protected] Tomaso PoggioMIT [email protected] Pierfrancesco BeneventanoMIT [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63732 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16622