Activation Outliers in Transformer Quantization: Reproduction, Statistical Analysis, and Deployment Tradeoffs

Pranav Kumar Kaliaperumal · 2026

Read Activation Outliers in Transformer Quantization: Reproduction, Statistical Analysis, and Deployment Tradeoffs by Pranav Kumar Kaliaperumal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04308.txt.

Description

Kaliaperumal reproduces a known transformer quantization failure and extends it at systems level. For BERT base on QNLI, global W8A8 quantization drops accuracy from 89.66 percent in full precision by more than 35 points. Activations grow heavy tailed with depth, kurtosis reaching 271 in late layers. Mixed precision nearly restores baseline accuracy, while percentile calibration fails to recover it. Large channels thus carry structured signal, not rare noise, so channel aware precision beats scalar clipping. The study thus warns that blunt cutoffs miss real signal.

Catalog record

CREATORPranav Kumar Kaliaperumal
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41740
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.04308

Browse the full library · About the Tower of Babel