LoopQ: Quantization for Recursive Transformers

Rui Fang, Hsi-Wen Chen, Ming-Syan Chen · 2026

Read LoopQ: Quantization for Recursive Transformers by Rui Fang, Hsi-Wen Chen, Ming-Syan Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16343.txt.

Introduction

Fang and three colleagues measure LoopQ against the strongest static post-training quantization baseline per comparison. Under W4A4 quantization it lifts five-task mean accuracy by 67.8 percent across four backbones. It cuts average relative perplexity by a conservative 75.2 percent across eight backbone-dataset pairs. They trace the damage to shifting distributions as blocks change roles, states across loop transitions, and errors piling up recursively. LoopQ answers with loop-aware activation scaling, cross-loop state alignment, and trajectory-aware calibration, keeping a shared quantized backbone with adaptations.

Catalog record

CREATORRui Fang, Hsi-Wen Chen, Ming-Syan Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65575
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16343

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LoopQ: Quantization for Recursive Transformers Rui Fang, Hsi-Wen Chen, Ming-Syan Chen Abstract Looped language models (LoopLMs) improve parameter efficiency by recursively reusing Transformer blocks, enabling deeper computation under a fixed model size. However, this reuse makes LoopLMs more fragile under post-training quantization (PTQ). We present the first systematic study of quantization in LoopLMs and identify three challenges: distribution shift across roles , state reuse across loop transitions , and recursive error accumulation . To address these challenges, we propose LoopQ , a loop-aware PTQ framework that preserves a shared quantized backbone while introducing lightweight adaptations. LoopQ combines activation scaling, selective transformation, cross-loop state alignment, and trajectory-aware optimization to reduce distributional mismatch within loops and error accumulation across loops. Experiments across seven benchmarks show that, under W4A4 quantization, LoopQ improves average downstream accuracy by 68.8% and reduces average perplexity by 87.7% compared with the strongest static PTQ baseline. 1 Introduction Quantization robustness [ 1 , 34 , 38 ] is critical for efficient LLM deployment on resource-constrained devices [ 39 ] . Looped language models (LoopLMs) [ 5 , 21 , 45 , 17 , 3 , 8 ] improve expressiveness under a fixed parameter budget by recursively reusing shared Transformer blocks , making them attractive for memory- and bandwidth-constrained settings [ 43 , 20 , 19 ] . However, existing LoopLMs mainly target full-precision inference, while most post-training quantization (PTQ) methods optimize layer-wise quantization error [ 37 , 2 , 22 , 18 , 14 , 35 , 32 ] without accounting for error propagation through recursively reused states. This limitation is particularly problematic for LoopLMs, which are substantially more sensitive to quantization than comparable non-loop architectures [ 40 ] . In our preliminary experiments, directly applying PTQ to LoopLMs leads to performance drops exceeding 50%. Under the same parameter budget, quantized non-loop LLMs can even outperform higher-precision LoopLMs. This suggests that the benefits of parameter reuse in full-precision settings do not automatically transfer to quantized regimes, where recursive reuse can amplify quantization-induced errors. Figure 1 : Left: LoopLMs exhibit loop-dependent activation drift (Challenge 1), error spikes at loop transitions (Challenge 2), and error accumulation across loops (Challenge 3). Right: LoopQ preserves a shared quantized backbone while adding lightweight loop-aware scaling, selective transformation, and cross-loop state alignment to reduce recursive quantization errors. Figure 1 illustrates this vulnerability through two coupled phenomena induced by quantizing recursive computation. First, the top-left panel shows that the same physical layer exhibits different normalized P99 activation ranges across loop passes, indicating that a single static quantization configuration cannot accommodate multiple loop-dependent regimes. Second, the bottom-left panel shows that quantization errors do not remain local. Instead, they grow as recursion proceeds and exhibit pronounced spikes at loop transitions, where the hidden state produced by one loop is passed to the next. These observations suggest that quantization in LoopLMs is governed by both per-layer distributional mismatch and recursive error accumulation. The root cause is that the same shared block is reused under different computational roles as recursion proceeds. A single physical layer therefore no longer behaves as a fixed feedforward position, but instead operates under stage-dependent roles across loops. This role variation leads to three key challenges, which we formalize through theoretical analysis: (1) Distribution Shift across Roles , where loop-dependent activation statistics cause scale and geometry mismatches under shared quantization; (2) State Reuse across Transitions , where the hidden state produced by one loop is reused as the input to the next, introducing transition inconsistency; and (3) Recursive Error Accumulation , where quantization errors propagate along the recurrent trajectory and are amplified by subsequent loop-dependent transformations. A naive solution is to assign loop-specific quantization configurations to all loops, thereby fully adapting to loop-dependent distributions. However, this breaks the shared-parameter structure of recursive computation and introduces substantial memory and deployment overhead, undermining the efficiency advantages of LoopLMs. We therefore propose LoopQ , a loop-aware post-training quantization framework that preserves a shared quantized backbone while introducing lightweight loop-dependent adaptations that affect less than 5% of transformation parameters, corresponding to less than 0.12% of total model parameters. As shown in the right panel of Figure 1 , Loop-aware Activation Scaling (brown block) uses loop-dependent scales to track magnitude drift, while Selective Loop-aware Transformation (blue block) selectively introduces loop-dependent transforms for sharing-sensitive modules identified by sharing-gap analysis , correcting residual geometry mismatch. To stabilize loop transitions, Cross-loop Transition Adapter (green block) aligns cross-loop hidden states with a lightweight adapter. Finally, guided by fixed-point convergence [ 17 ] , Trajectory-Aware Calibration performs progressive trajectory-aware calibration under the true recurrent dynamics to suppress trajectory-level error accumulation. Our contributions are summarized as follows. (i) We provide the first systematic study of quantization in LoopLMs, identifying three foundational challenges in recursive computation: distribution shift across roles, state reuse across transitions, and recursive error accumulation. (ii) We propose LoopQ, a loop-aware post-training quantization framework that preserves a shared quantized backbone while introducing lightweight loop-dependent adaptations, including activation scaling, selective transformation, cross-loop state alignment, and trajectory-aware optimization. (iii) Experiments on seven benchmarks show that LoopQ outperforms five PTQ baselines, improving mean downstream accuracy by 68.8% and reducing average perplexity by 87.7% under W4A4 quantization. 2 Problem Formulation We first introduce the LoopLM architecture, followed by the post-training quantization setting. LoopLM. LoopLM is a recursive Transformer composed of a shared stack of L layers, applied iteratively for T loops. Let \mathbf{X} denote the input, and let f_{\ell}(\cdot;\theta_{\ell}) represent the \ell -th layer. We denote the hidden state at layer \ell in loop t as \mathbf{H}_{t,\ell}\in\mathbb{R}^{n\times d} , where each row corresponds to a token representation. The initial state is \mathbf{H}_{0,0}=\mathrm{Embed}(\mathbf{X}) . The loop computation is defined as \mathbf{H}_{t,\ell}=f_{\ell}(\mathbf{H}_{t,\ell-1};\theta_{\ell}),\hskip 18.49988pt\mathbf{H}_{t+1,0}=\mathbf{H}_{t,L}, (1) for t\in\{0,\ldots,T-1\} and \ell\in\{1,\ldots,L\} . Let \mathbf{H}_{t}:=\mathbf{H}_{t,0} . Each loop can be summarized by F:=f_{L}\circ\cdots\circ f_{1} , yielding \mathbf{H}_{t+1}=F(\mathbf{H}_{t}) , and the final output is \mathbf{Z}=\mathrm{Proj}(\mathbf{H}_{T}) . For analytical clarity, we express f_{\ell}(\cdot;\theta_{\ell}) using a single weight matrix \mathbf{W}_{\ell} , i.e., \mathbf{H}_{t,\ell}=\mathbf{H}_{t,\ell-1}\mathbf{W}_{\ell}^{\top} . This abstraction naturally extends to general Transformer blocks with multiple weight matrices. Post-training Quantization on LoopLMs. We consider a standard post-training quantization (PTQ) setting, where each layer f_{\ell} is approximated by a quantized counterpart \widetilde{f}_{\ell} with quantized weights \hat{\theta}_{\ell} . Since model weights and activations often exhibit imbalanced distributions, such as heavy tails or channel-wise variance disparity, direct quantization is suboptimal because quantization levels are inefficiently allocated over the dynamic range. To address this, we employ an invertible transformation \mathbf{P}_{\ell} to reshape the representation while preserving the original function. For each layer \ell , the full-precision mapping can be equivalently written as \mathbf{H}_{t,\ell}=(\mathbf{H}_{t,\ell-1}\mathbf{P}_{\ell})(\mathbf{W}_{\ell}\mathbf{P}_{\ell}^{-\top})^{\top}. (2) Accordingly, quantized inference is performed as \widetilde{\mathbf{H}}_{t,\ell}=Q_{a}(\widetilde{\mathbf{H}}_{t,\ell-1}\mathbf{P}_{\ell};c_{\ell})\,Q_{w}(\mathbf{W}_{\ell}\mathbf{P}_{\ell}^{-\top})^{\top},\hskip 18.49988pt\widetilde{\mathbf{H}}_{t+1,0}=\widetilde{\mathbf{H}}_{t,L}. (3) Here, Q_{w} and Q_{a} denote weight and activation quantization, respectively. For an activation element x at layer \ell , symmetric uniform quantization is defined as Q_{a}(x;c_{\ell})=c_{\ell}\cdot\mathrm{clip}\!\left(\left\lfloor x/c_{\ell}\right\rceil,q_{\min},q_{\max}\right), (4) where c_{\ell} determines the spacing between quantization levels, and q_{\min} and q_{\max} denote the minimum and maximum representable integer levels, respectively. Compared to conventional quantization, quantizing LoopLMs introduces two distinct sources of mismatch: (i) loop-wise variation in activation distributions within each shared layer, where the quantization parameters (\mathbf{P}_{\ell},c_{\ell}) are fixed across loops while activations vary across loops; and (ii) recursive evolution of hidden states, where \mathbf{H}_{t,\ell} changes across loops, leading to compounded distribution

Read the complete text