Scaling Laws for Floating–Point Quantization Training
Xingwu Sun, Shuaipeng Li, Ruobing Xie, Weidong Han, Kan Wu, Zhen Yang, Yixing Li, An Wang, Shuai Li, Jinbao Xue, Yu Cheng, Yangyu Tao, Zhanhui Kang, Chengzhong Xu, Di Wang, Jie Jiang · 2025
Read Scaling Laws for Floating–Point Quantization Training by Xingwu Sun, Shuaipeng Li, Ruobing Xie, Weidong Han, Kan Wu, Zhen Yang, Yixing Li, An Wang, Shuai Li, Jinbao Xue, Yu Cheng, Yangyu Tao, Zhanhui Kang, Chengzhong Xu, Di Wang, Jie Jiang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-02423.txt.
Description
Xingwu Sun and co-authors map the scaling laws of training language models in low-precision floating point. Cutting number precision slashes training and serving costs, and production shops already train this way. Old scaling laws, though, were fit to integer quantization and miss how floats divide their bits. Their 2025 study varies each ingredient in turn.
What gets quantized, how many bits go to exponent versus mantissa, and how coarsely scale factors are shared all move the final loss. Exponent bits buy slightly more quality than mantissa bits, and the team tabulates the best split at each total width. A unified law then predicts loss across the whole design space. Model trainers get a recipe card for cheap precision without the usual trial and error.
Catalog record
| CREATOR | Xingwu Sun, Shuaipeng Li, Ruobing Xie, Weidong Han, Kan Wu, Zhen Yang, Yixing Li, An Wang, Shuai Li, Jinbao Xue, Yu Cheng, Yangyu Tao, Zhanhui Kang, Chengzhong Xu, Di Wang, Jie Jiang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 71 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-02423