SimpleGPT: Improving GPT via A Simple Normalization Strategy
Marco Chen, Xianbiao Qi, Yelin He, Jiaquan Ye, Rong Xiao · 2026
Read SimpleGPT: Improving GPT via A Simple Normalization Strategy by Marco Chen, Xianbiao Qi, Yelin He, Jiaquan Ye, Rong Xiao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01212.txt.
Description
Marco Chen and co-authors lift GPT training with SimpleNorm, a plain scale rule from second-order geometry. They link net design, act scale, the Hessian map, and top safe learn rate. SimpleNorm steadies mid act scales by build, and Hessian math shows it shrinks the Hessian's top singular value a lot, so bigger safe learn rates hold. Tests on GPT nets from 1B to 8B weights show far bigger tolerated learn rates, firm trains, and better scores than baselines, with lower train loss at 7B scale.
Catalog record
| CREATOR | Marco Chen, Xianbiao Qi, Yelin He, Jiaquan Ye, Rong Xiao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64880 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.01212