A Geometric Analysis of Sign-Magnitude Asymmetry in a ReLU + RMSNorm Block under Ternary Quantization
Dong Lei · 2026
Read A Geometric Analysis of Sign-Magnitude Asymmetry in a ReLU + RMSNorm Block under Ternary Quantization by Dong Lei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18933.txt.
Description
Lei Dong explains why pre norm transformers cope well with three level weights. Sign flips yield about two and three quarter times the sideways energy of equal size shifts. ReLU tilts hidden space in one direction, and the norm layer lays that tilt bare. Ternary error keeps sign, slips through ReLU near unchanged, and real sensitivity stems from rare large features.
Catalog record
| CREATOR | Dong Lei |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 133633 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.18933