MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration

Lianhai Ren, Yucheng Ding, Xiao Liu, Qianxiao Li, Peng Cheng, Yeyun Gong · 2026

Read MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration by Lianhai Ren, Yucheng Ding, Xiao Liu, Qianxiao Li, Peng Cheng, Yeyun Gong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01734.txt.

Description

Lianhai Ren and colleagues study training instability in language model pretraining, where sudden gradient explosions waste compute. In a NanoGPT model scaled by muP, they identify two warning signs before collapse. One is decline in weight stable rank, and the other is growing alignment between adjacent layer Jacobians. They prove both conditions together cause gradient norms to grow exponentially with network depth. Their MSign periodically uses matrix sign operations to bring back stable rank, preventing failures from five million to three billion parameter models with under seven percent compute.

Catalog record

CREATORLianhai Ren, Yucheng Ding, Xiao Liu, Qianxiao Li, Peng Cheng, Yeyun Gong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80818
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.01734

Browse the full library · About the Tower of Babel