On the Spectral Flattening of Quantized Embeddings
Junlin Huang, Wenyi Fang, Zhenheng Tang, Yuxin Wang, Xueze Kang, Yang Zheng, Bo Li, Xiaowen Chu · 2026
Read On the Spectral Flattening of Quantized Embeddings by Junlin Huang, Wenyi Fang, Zhenheng Tang, Yuxin Wang, Xueze Kang, Yang Zheng, Bo Li, Xiaowen Chu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-00969.txt.
Description
Junlin Huang and colleagues trace low-precision training crashes to a clash between coarse quantization and the spectra of data. They link Zipf-style patterns to matrix theory, proving that power decay in embedding singular values is needed for meaning to be encoded. Uniform quantization sets a noise floor that chops the spectral tail, flattens spectra, and lifts the rank of representations. Tests on GPT-2 and TinyLlama bear out the slide toward representational collapse. In short, keeping spectral shape intact is a must for low-bit training.
Catalog record
| CREATOR | Junlin Huang, Wenyi Fang, Zhenheng Tang, Yuxin Wang, Xueze Kang, Yang Zheng, Bo Li, Xiaowen Chu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86016 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.00969