A Geometric Analysis of Sign-Magnitude Asymmetry in a ReLU + RMSNorm Block under Ternary Quantization

Dong Lei · 2026

Read A Geometric Analysis of Sign-Magnitude Asymmetry in a ReLU + RMSNorm Block under Ternary Quantization by Dong Lei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-18933.txt.

Description

Lei Dong explains why pre norm transformers cope well with three level weights. Sign flips yield about two and three quarter times the sideways energy of equal size shifts. ReLU tilts hidden space in one direction, and the norm layer lays that tilt bare. Ternary error keeps sign, slips through ReLU near unchanged, and real sensitivity stems from rare large features.

Catalog record

CREATORDong Lei
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE133633
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.18933

Browse the full library · About the Tower of Babel