SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Training Conference: Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2; March 22–26, 2026; Pittsburgh, PA, USA Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’26), March 22–26, 2026, Pittsburgh, PA, USA DOI: 10.1145/3779212.3790223 ISBN: 979-8-4007-2359-9/2026/03 CCS: Computer systems organization Architectures CCS: Computing methodologies Neural networks

Yunjie Pan, Yongyi Yang, Hanmei Yang, Scott Mahlke · 2026

Read SNIP: An Adaptive Mixed Precision Framework for Subbyte Large Language Model Training Conference: Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2; March 22–26, 2026; Pittsburgh, PA, USA Proceedings of the 31st ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’26), March 22–26, 2026, Pittsburgh, PA, USA DOI: 10.1145/3779212.3790223 ISBN: 979-8-4007-2359-9/2026/03 CCS: Computer systems organization Architectures CCS: Computing methodologies Neural networks by Yunjie Pan, Yongyi Yang, Hanmei Yang, Scott Mahlke (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-01410.txt.

Description

Yunjie Pan and co-authors introduce SNIP, a mixed-precision trainer for language model pretraining at subbyte precision. Uniform GEMM precision or fragile heuristics cause poor convergence and instability today. SNIP gathers activation, gradient, and optimizer statistics to judge precision loss against model quality. Forward loss divergence and backward weight divergence feed an Integer Linear Programming task setting per-layer precision for minimum quality loss. Tests on Llama-like models at 1B, 3B, 7B, and 70B show SNIP topping baselines, cutting FLOPs up to 80% while holding quality with little overhead.

Catalog record

CREATORYunjie Pan, Yongyi Yang, Hanmei Yang, Scott Mahlke
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67105
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.01410

Browse the full library · About the Tower of Babel