Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition
Ziyang Liu · 2026
Read Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition by Ziyang Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18128.txt.
Introduction
Liu studies why extreme 4-bit compression wrecks a small SwiGLU language model. Under naive rounding, validation perplexity jumps from about 24 to over 1,700. A training-time fix called Depth Registers tames the residual stream and cuts that figure to 119. Pairing it with SmoothQuant brings it under 40. A careful split of the damage shows the surviving error lives mainly in one internal matrix, w2, which rotations alone cannot bound.
Catalog record
| CREATOR | Ziyang Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47824 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.18128
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition Ziyang Liu Abstract We study post-training W4A4 quantization in a controlled 300M-parameter SwiGLU decoder-only language model trained on 5B tokens of FineWeb-Edu, and ask which input-activation sites dominate the error. Naive round-to-nearest W4A4 collapses validation perplexity from FP16 23.6 23.6 to 1727 1727 . A simple residual-axis training-time intervention — Depth Registers with a register-magnitude hinge loss ( DR+sink ) — reduces this to 119 119 ( ∼ 14 × \sim 14\times ) at matched FP16 PPL and matched zero-shot capacity, and composes with SmoothQuant to 39.9 39.9 PPL. The residual ≈ 2 \approx 2 PPL gap to FP16 is the diagnostic core. We decompose W4A4 damage by input-activation site: the five trainable linears in a SwiGLU block split into residual-axis readers ( qkv , w1 , w3 ) and block-internal generators ( o_proj , w2 ). Elementary norm arguments show residual-axis magnitude control bounds readers tightly but leaves w2 ’s bilinear input bounded only by the trivial product of factor bounds; empirically, DR+sink collapses reader kurtosis while leaving generators essentially unchanged, and the reader-rescued W4A4 residue is flat at ≈ 0.28 \approx 0.28 nats across three matched checkpoints with Δ remove ( w2 ) \Delta\mathrm{remove}(\texttt{w2}) dominating. We present DR+sink as a training-time probe rather than a deployment proposal: a post-hoc alternative (Per-Linear QuaRot) nearly matches it on the reader axis. Full QuaRot — adding online per-head value Hadamard plus online w2 -input rotation — does not close the gap either, directly testing the prediction that orthogonal rotation cannot bound the bilinear SwiGLU tail. Claims are specific to our 300M, 5B-token, single-seed setting, and our experiments do not isolate the partition from the hinge. 1 Introduction W4A4 post-training quantization (4-bit weights, 4-bit activations) is the leading route to 2–3 \times inference throughput and memory savings on commodity GPUs. SwiGLU-based language models — LLaMA ( Touvron et al. 2023 ) , Mistral ( Jiang et al. 2023 ) , Qwen ( Bai et al. 2023 ) — are reported to lose one to three orders of magnitude in validation perplexity under naive round-to-nearest W4A4; we reproduce this on a 300M-parameter SwiGLU decoder-only model trained from scratch on 5B FineWeb-Edu tokens (PPL 23.6\to 1727 ). A family of calibration-based remedies — SmoothQuant ( Xiao et al. 2023 ) , QuaRot ( Ashkboos et al. 2024 ) , AWQ ( Lin et al. 2024 ) , FlatQuant ( Sun et al. 2025 ) — recovers most of this gap by rescaling, rotating, or protecting specific linear layers. Each choice encodes an implicit hypothesis about which linears drive the damage and why . Prior work catalogs the phenomenology but does not separate the damage by input-activation site. Observational studies document a few hidden-state channels carrying very large magnitudes ( Dettmers et al. 2022 ; Sun et al. 2024 ) , an “attention sink” token emitting disproportionate values ( Xiao et al. 2024 ) , and GLU-family FFNs whose elementwise product amplifies channel-wise spikes ( Yang et al. 2025 ) . These say where outliers appear; they do not say which linears’ input activations are load-bearing for W4A4 damage, nor whether those inputs are reachable from the residual axis. We offer a clean operator-level account. In a SwiGLU block, five input-activation sites can be grouped by the source of their input: • Readers receive the residual stream after RMSNorm : qkv , w1 , w3 . Their input magnitude is directly bounded by any intervention that shapes the residual. • Generators receive a function of the block’s internal computation: o_proj ’s input is the attention-weighted mixture \sum_{j}\alpha_{j}V(\tilde{x}_{j}) — linear in the value projections conditional on the softmax weights \alpha_{j} , which themselves depend on \tilde{x} ; w2 ’s input is the SwiGLU product \silu(\texttt{w1}(\tilde{x}))\odot\texttt{w3}(\tilde{x}) , which is bilinear in the residual. Figure 1 : The five input-activation sites of a SwiGLU block. Readers ( qkv , w1 , w3 ; teal) ingest \RMSNorm of the residual stream; generators ( o_proj , w2 ; amber) ingest block-internal quantities — the softmax \cdot V aggregate and the SwiGLU bilinear product. The algebra ( Section 3 ) predicts that a norm-constrained residual-axis intervention bounds reader inputs tightly, bounds o_proj ’s input only up to \left\|W_{V}\right\|_{\mathrm{op}} and the softmax weights the attention assigns to outlier positions, and does not bound the SwiGLU product’s heavy tail beyond a trivial product of bounds. We propose a residual-axis training method that exploits this decomposition directly: Depth Registers with a register-magnitude hinge loss (DR+sink) . DR+sink partitions the residual stream into register and non-register dimensions and applies a hinge penalty pulling the maximum register magnitude below a fixed target ( Section 3.3 ). On a 300M \times 5B FineWeb-Edu checkpoint, DR+sink reduces naive-RTN W4A4 perplexity from \mathbf{1727} to \mathbf{119} ( \boldsymbol{\sim 14\times} ) at matched FP16 PPL ( 23.6 ) and matched zero-shot capacity (three 1k-example probes, \pm 1.5 pp), and composes with SmoothQuant to reach 39.9 PPL (from 57.6 without the probe). Operator-level evidence aligns with the decomposition: DR+sink drives reader kurtosis to near-Gaussian ( qkv : 35.9\to 2.85 ) and leaves generator kurtosis essentially unchanged ( w2 : 1921\to 1860 ) across three matched checkpoints. The same separation shows up directly in W4A4 damage. We report skip-ablation observables in excess NLL (conditional-on-rest-quantized, not additive): \Delta\mathrm{NLL}(\text{naive}) , \Delta\mathrm{NLL}(\text{skip-readers}) — the reader-rescued residue — and single-site skip-removals ( Section 4.2 ). The reader-rescued residue is flat across all three matched checkpoints at \approx 0.28 nats ( \approx 31 PPL). A post-hoc alternative, Per-Linear QuaRot (independent random Hadamard rotations per linear input), substitutes for DR+sink on the reader axis almost exactly ( 25.5 / 25.4 / 25.4 W4A4 PPL, spread \leq 0.1 ). We therefore do not claim DR+sink is the uniquely best practical recipe for W4A4. Our claim is narrower: DR+sink is a training-time residual-axis probe that makes the reader/generator split experimentally visible, composes with residual-axis post-hoc scaling (SmoothQuant: 57.6\to 39.9 PPL, a reduction Per-Linear QuaRot does not produce), and produces an interpretable named partition rather than a structureless rotation. Full QuaRot (adding online per-head Hadamard on attention values and online orthogonal rotation on w2 ’s input) lands at 26.0 / 26.1 / 26.0 PPL ( \Delta\mathrm{NLL}\approx 0.09 nats), marginally worse than per-Linear QuaRot despite the online FFN-internal w2 rotation — the direct post-hoc test of 3.4 : in our setting, orthogonal rotation does not close the bilinear SwiGLU residue. Contributions. 1. An operator-level decomposition of the five input-activation sites in a SwiGLU block into residual-axis readers and block-internal generators , with elementary norm arguments bounding reader inputs by the residual, bounding o_proj ’s input up to a softmax- and W_{V} -dependent constant, and leaving w2 ’s bilinear product bounded only by the trivial product of factor bounds ( Section 3 ). 2. A residual-axis training method DR+sink (Depth Registers + register-magnitude hinge) that reduces naive-RTN W4A4 PPL from 1727 to 119 ( \sim 14\times ) at matched FP16 PPL ( 23.6 ) and matched zero-shot capacity (three 1k-example probes, \pm 1.5 pp) on a 300M \times 5B checkpoint, and composes with SmoothQuant ( 57.6\to 39.9 PPL). Operator-level: reader kurtosis collapses ( qkv : 35.9\to 2.85 ) while generator kurtosis is essentially