Scaling Laws for Floating–Point Quantization Training

Xingwu Sun, Shuaipeng Li, Ruobing Xie, Weidong Han, Kan Wu, Zhen Yang, Yixing Li, An Wang, Shuai Li, Jinbao Xue, Yu Cheng, Yangyu Tao, Zhanhui Kang, Chengzhong Xu, Di Wang, Jie Jiang · 2025

Read Scaling Laws for Floating–Point Quantization Training by Xingwu Sun, Shuaipeng Li, Ruobing Xie, Weidong Han, Kan Wu, Zhen Yang, Yixing Li, An Wang, Shuai Li, Jinbao Xue, Yu Cheng, Yangyu Tao, Zhanhui Kang, Chengzhong Xu, Di Wang, Jie Jiang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-02423.txt.

Description

Xingwu Sun and co-authors map the scaling laws of training language models in low-precision floating point. Cutting number precision slashes training and serving costs, and production shops already train this way. Old scaling laws, though, were fit to integer quantization and miss how floats divide their bits. Their 2025 study varies each ingredient in turn.

What gets quantized, how many bits go to exponent versus mantissa, and how coarsely scale factors are shared all move the final loss. Exponent bits buy slightly more quality than mantissa bits, and the team tabulates the best split at each total width. A unified law then predicts loss across the whole design space. Model trainers get a recipe card for cheap precision without the usual trial and error.

Catalog record

CREATORXingwu Sun, Shuaipeng Li, Ruobing Xie, Weidong Han, Kan Wu, Zhen Yang, Yixing Li, An Wang, Shuai Li, Jinbao Xue, Yu Cheng, Yangyu Tao, Zhanhui Kang, Chengzhong Xu, Di Wang, Jie Jiang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-02423

Browse the full library · About the Tower of Babel