SimpleGPT: Improving GPT via A Simple Normalization Strategy

Marco Chen, Xianbiao Qi, Yelin He, Jiaquan Ye, Rong Xiao · 2026

Read SimpleGPT: Improving GPT via A Simple Normalization Strategy by Marco Chen, Xianbiao Qi, Yelin He, Jiaquan Ye, Rong Xiao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01212.txt.

Description

Marco Chen and co-authors lift GPT training with SimpleNorm, a plain scale rule from second-order geometry. They link net design, act scale, the Hessian map, and top safe learn rate. SimpleNorm steadies mid act scales by build, and Hessian math shows it shrinks the Hessian's top singular value a lot, so bigger safe learn rates hold. Tests on GPT nets from 1B to 8B weights show far bigger tolerated learn rates, firm trains, and better scores than baselines, with lower train loss at 7B scale.

Catalog record

CREATORMarco Chen, Xianbiao Qi, Yelin He, Jiaquan Ye, Rong Xiao
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE64880
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.01212

Browse the full library · About the Tower of Babel