Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition

Ziyang Liu · 2026

Read Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition by Ziyang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18128.txt.

Introduction

Liu studies why extreme 4-bit compression wrecks a small SwiGLU language model. Under naive rounding, validation perplexity jumps from about 24 to over 1,700. A training-time fix called Depth Registers tames the residual stream and cuts that figure to 119. Pairing it with SmoothQuant brings it under 40. A careful split of the damage shows the surviving error lives mainly in one internal matrix, w2, which rotations alone cannot bound.

Catalog record

CREATORZiyang Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47824
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18128

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition Ziyang Liu Abstract We study post-training W4A4 quantization in a controlled 300M-parameter SwiGLU decoder-only language model trained on 5B tokens of FineWeb-Edu, and ask which input-activation sites dominate the error. Naive round-to-nearest W4A4 collapses validation perplexity from FP16 23.6 23.6 to 1727 1727 . A simple residual-axis training-time intervention — Depth Registers with a register-magnitude hinge loss ( DR+sink ) — reduces this to 119 119 ( ∼ 14 × \sim 14\times ) at matched FP16 PPL and matched zero-shot capacity, and composes with SmoothQuant to 39.9 39.9 PPL. The residual ≈ 2 \approx 2 PPL gap to FP16 is the diagnostic core. We decompose W4A4 damage by input-activation site: the five trainable linears in a SwiGLU block split into residual-axis readers ( qkv , w1 , w3 ) and block-internal generators ( o_proj , w2 ). Elementary norm arguments show residual-axis magnitude control bounds readers tightly but leaves w2 ’s bilinear input bounded only by the trivial product of factor bounds; empirically, DR+sink collapses reader kurtosis while leaving generators essentially unchanged, and the reader-rescued W4A4 residue is flat at ≈ 0.28 \approx 0.28 nats across three matched checkpoints with Δ ​ remove ​ ( w2 ) \Delta\mathrm{remove}(\texttt{w2}) dominating. We present DR+sink as a training-time probe rather than a deployment proposal: a post-hoc alternative (Per-Linear QuaRot) nearly matches it on the reader axis. Full QuaRot — adding online per-head value Hadamard plus online w2 -input rotation — does not close the gap either, directly testing the prediction that orthogonal rotation cannot bound the bilinear SwiGLU tail. Claims are specific to our 300M, 5B-token, single-seed setting, and our experiments do not isolate the partition from the hinge. 1 Introduction W4A4 post-training quantization (4-bit weights, 4-bit activations) is the leading route to 2–3 \times inference throughput and memory savings on commodity GPUs. SwiGLU-based language models — LLaMA ( Touvron et al. 2023 ) , Mistral ( Jiang et al. 2023 ) , Qwen ( Bai et al. 2023 ) — are reported to lose one to three orders of magnitude in validation perplexity under naive round-to-nearest W4A4; we reproduce this on a 300M-parameter SwiGLU decoder-only model trained from scratch on 5B FineWeb-Edu tokens (PPL 23.6\to 1727 ). A family of calibration-based remedies — SmoothQuant ( Xiao et al. 2023 ) , QuaRot ( Ashkboos et al. 2024 ) , AWQ ( Lin et al. 2024 ) , FlatQuant ( Sun et al. 2025 ) — recovers most of this gap by rescaling, rotating, or protecting specific linear layers. Each choice encodes an implicit hypothesis about which linears drive the damage and why . Prior work catalogs the phenomenology but does not separate the damage by input-activation site. Observational studies document a few hidden-state channels carrying very large magnitudes ( Dettmers et al. 2022 ; Sun et al. 2024 ) , an “attention sink” token emitting disproportionate values ( Xiao et al. 2024 ) , and GLU-family FFNs whose elementwise product amplifies channel-wise spikes ( Yang et al. 2025 ) . These say where outliers appear; they do not say which linears’ input activations are load-bearing for W4A4 damage, nor whether those inputs are reachable from the residual axis. We offer a clean operator-level account. In a SwiGLU block, five input-activation sites can be grouped by the source of their input: • Readers receive the residual stream after RMSNorm : qkv , w1 , w3 . Their input magnitude is directly bounded by any intervention that shapes the residual. • Generators receive a function of the block’s internal computation: o_proj ’s input is the attention-weighted mixture \sum_{j}\alpha_{j}V(\tilde{x}_{j}) — linear in the value projections conditional on the softmax weights \alpha_{j} , which themselves depend on \tilde{x} ; w2 ’s input is the SwiGLU product \silu(\texttt{w1}(\tilde{x}))\odot\texttt{w3}(\tilde{x}) , which is bilinear in the residual. Figure 1 : The five input-activation sites of a SwiGLU block. Readers ( qkv , w1 , w3 ; teal) ingest \RMSNorm of the residual stream; generators ( o_proj , w2 ; amber) ingest block-internal quantities — the softmax \cdot V aggregate and the SwiGLU bilinear product. The algebra ( Section 3 ) predicts that a norm-constrained residual-axis intervention bounds reader inputs tightly, bounds o_proj ’s input only up to \left\|W_{V}\right\|_{\mathrm{op}} and the softmax weights the attention assigns to outlier positions, and does not bound the SwiGLU product’s heavy tail beyond a trivial product of bounds. We propose a residual-axis training method that exploits this decomposition directly: Depth Registers with a register-magnitude hinge loss (DR+sink) . DR+sink partitions the residual stream into register and non-register dimensions and applies a hinge penalty pulling the maximum register magnitude below a fixed target ( Section 3.3 ). On a 300M \times 5B FineWeb-Edu checkpoint, DR+sink reduces naive-RTN W4A4 perplexity from \mathbf{1727} to \mathbf{119} ( \boldsymbol{\sim 14\times} ) at matched FP16 PPL ( 23.6 ) and matched zero-shot capacity (three 1k-example probes, \pm 1.5 pp), and composes with SmoothQuant to reach 39.9 PPL (from 57.6 without the probe). Operator-level evidence aligns with the decomposition: DR+sink drives reader kurtosis to near-Gaussian ( qkv : 35.9\to 2.85 ) and leaves generator kurtosis essentially unchanged ( w2 : 1921\to 1860 ) across three matched checkpoints. The same separation shows up directly in W4A4 damage. We report skip-ablation observables in excess NLL (conditional-on-rest-quantized, not additive): \Delta\mathrm{NLL}(\text{naive}) , \Delta\mathrm{NLL}(\text{skip-readers}) — the reader-rescued residue — and single-site skip-removals ( Section 4.2 ). The reader-rescued residue is flat across all three matched checkpoints at \approx 0.28 nats ( \approx 31 PPL). A post-hoc alternative, Per-Linear QuaRot (independent random Hadamard rotations per linear input), substitutes for DR+sink on the reader axis almost exactly ( 25.5 / 25.4 / 25.4 W4A4 PPL, spread \leq 0.1 ). We therefore do not claim DR+sink is the uniquely best practical recipe for W4A4. Our claim is narrower: DR+sink is a training-time residual-axis probe that makes the reader/generator split experimentally visible, composes with residual-axis post-hoc scaling (SmoothQuant: 57.6\to 39.9 PPL, a reduction Per-Linear QuaRot does not produce), and produces an interpretable named partition rather than a structureless rotation. Full QuaRot (adding online per-head Hadamard on attention values and online orthogonal rotation on w2 ’s input) lands at 26.0 / 26.1 / 26.0 PPL ( \Delta\mathrm{NLL}\approx 0.09 nats), marginally worse than per-Linear QuaRot despite the online FFN-internal w2 rotation — the direct post-hoc test of 3.4 : in our setting, orthogonal rotation does not close the bilinear SwiGLU residue. Contributions. 1. An operator-level decomposition of the five input-activation sites in a SwiGLU block into residual-axis readers and block-internal generators , with elementary norm arguments bounding reader inputs by the residual, bounding o_proj ’s input up to a softmax- and W_{V} -dependent constant, and leaving w2 ’s bilinear product bounded only by the trivial product of factor bounds ( Section 3 ). 2. A residual-axis training method DR+sink (Depth Registers + register-magnitude hinge) that reduces naive-RTN W4A4 PPL from 1727 to 119 ( \sim 14\times ) at matched FP16 PPL ( 23.6 ) and matched zero-shot capacity (three 1k-example probes, \pm 1.5 pp) on a 300M \times 5B checkpoint, and composes with SmoothQuant ( 57.6\to 39.9 PPL). Operator-level: reader kurtosis collapses ( qkv : 35.9\to 2.85 ) while generator kurtosis is essentially

Read the complete text