Training Deep Learning Models with Norm-Constrained LMOs

Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, Volkan Cevher · 2025

Read Training Deep Learning Models with Norm-Constrained LMOs by Thomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, Volkan Cevher (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-07529.txt.

Description

Thomas Pethick and five co-authors studied in 2025 training methods that steer each update by solving a small linear problem inside a norm ball. This oracle-based family adapts steps to the shape of the problem, and the authors show it can even serve on tasks with no explicit constraint. Their rule turns out to unify several known optimizers under one frame. A chosen norm for deep networks brings a bonus: settings transfer across model sizes.

Their algorithm, Scion, trained a small GPT model much faster than standard practice without leaning on Adam. It also uses little memory, holding one copy of the weights and one of the gradients, both storable at half precision. Code is public. The authors pitch the method at labs that need strong training on modest hardware. Geometry-aware steps, they argue, deserve a larger place in deep learning.

Catalog record

CREATORThomas Pethick, Wanyun Xie, Kimon Antonakopoulos, Zhenyu Zhu, Antonio Silveti-Falls, Volkan Cevher
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-07529

Browse the full library · About the Tower of Babel