Structural Sensitivity in Compressed Transformers: Error Propagation, Lyapunov Stability, and Formally Verified Bounds Journal: Neural Networks

Abhinaba Basu · 2026

Read Structural Sensitivity in Compressed Transformers: Error Propagation, Lyapunov Stability, and Formally Verified Bounds Journal: Neural Networks by Abhinaba Basu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20991.txt.

Introduction

Basu and co-authors measure how compression errors move through transformer layers. Compressing weights makes large language models cheaper to run, but each layer adds error. They define a ratio, rho, of output error to input error at each layer. Below one, a layer absorbs error; above one, it grows. On six transformers, later rho values predict how much representations drift. This explains why early layer compression hurts more. They also rank layers for removal with two forward passes, beating a known method at lower perplexity.

Catalog record

CREATORAbhinaba Basu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55732
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20991

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Structural Sensitivity in Compressed Transformers: Error Propagation, Lyapunov Stability, and Formally Verified Bounds Journal: Neural Networks Abhinaba Basu Abstract A single matrix out of 468 in GPT-2 Small can increase perplexity by 20,000 × 20{,}000\times when compressed—revealing that transformer compression sensitivity spans five orders of magnitude and is dominated by a few critical components. We map this sensitivity landscape across five architectures (117M–8B parameters), finding a consistent hierarchy: early-layer MLP up-projections are catastrophically sensitive while value projections compress nearly for free. This hierarchy is stable across compression levels, evaluation scales (2K–51K tokens), and datasets (WikiText-103, C4). Using Lyapunov stability theory, we show that residual connections—not LayerNorm alone—contract compression errors by growing the hidden state faster than the error. Error contraction is necessary but not sufficient: Qwen3-8B ( ρ max = 1.33 \rho_{\max}=1.33 ) collapses at 42,922 × 42{,}922\times degradation while Mistral-7B ( ρ max = 1.14 \rho_{\max}=1.14 ) degrades only 11 × 11\times , but the fully-contracting GPT-2 Small ( ρ max = 0.96 \rho_{\max}=0.96 ) degrades 120 × 120\times while the hybrid LFM2-2.6B degrades only 7 × 7\times . Compression robustness depends on both stability and architecture-specific redundancy. Ten machine-checked Lean 4 theorems formalize per-matrix error bounds with no sorry markers; all bounds produce zero violations across 14,040+ configurations. 1 Introduction Large language model inference is increasingly bottlenecked by memory bandwidth rather than compute [ 18 , 23 ] . This has motivated a rich literature on model compression—pruning [ 6 , 24 ] , quantization [ 7 , 4 ] , and low-rank factorization [ 11 ] —which has produced impressive practical results. However, a fundamental question remains underexplored: how do compression errors propagate through transformer layers, and what structural properties of the architecture govern this propagation? Understanding error propagation is critical for several reasons. First, practitioners need to know which components are safe to compress and which are not, without exhaustive empirical sweeps on every new model. Second, formal guarantees on output quality enable principled cost–quality trade-offs in deployment. Third, the mechanisms that cause certain components to be catastrophically sensitive to compression reveal fundamental aspects of transformer computation that are of independent scientific interest. Contributions. We make four contributions: 1. Structural sensitivity mapping (§ 5.1 ). We map compression sensitivity at per-matrix granularity across GPT-2 Small (468 matrices), GPT-2 Medium, and Mistral-7B. The sensitivity hierarchy ( mlp_fc \gg Q > K > mlp_proj > attn_proj > V) spans five orders of magnitude and is stable across compression levels, evaluation scales (2K–51K tokens), and datasets (WikiText-103, C4). Compressing layer 0’s MLP up-projection alone—one matrix out of 468—increases perplexity by 20{,}000\times . 2. Lyapunov stability analysis of error propagation (§ 5.2 ). We show that residual connections—not LayerNorm alone—contract compression errors by growing the hidden state faster than the error ( \rho_{\max}=0.96 on GPT-2 Small). This yields a geometric series bound 23\times tighter than the naive linear bound and explains why compression errors do not accumulate catastrophically through 12+ layers. 3. Cross-architecture validation: stability is necessary but not sufficient (§ 5.3 ). Validating across five architectures (117M–8B), we find that among similar architectures, higher \rho_{\max} correlates with worse compression outcomes (Qwen3-8B at 42{,}922\times vs. Mistral-7B at 11\times ), but across architecture families, redundancy dominates (LFM2-2.6B degrades only 7\times despite \rho_{\max}=1.12 , while the fully-contracting GPT-2 Small degrades 120\times ). 4. Formally verified per-matrix error bounds (§ 2 , § 3 ). Ten machine-checked Lean 4 theorems formalize per-matrix error bounds for three composable approximation stages, the composition theorem, and the Lyapunov contraction bound—with no sorry markers. All bounds produce zero violations across 14,040+ GPT-2 configurations, confirming that the formal guarantees on individual matrix multiplications are conservative when embedded in the full nonlinear forward pass. Our compression results (§ 5.4 ) demonstrate the practical value of the analysis. The greedy allocation (24% FLOPS savings at +27% perplexity) is not state-of-the-art in absolute compression quality; the contribution is the structural characterization that explains why non-uniform allocation is essential and which components to protect. Figure 1 : Overview. (a) Three composable approximation stages, each with machine-checked error bounds. (b) Lyapunov V(t) trajectories across five architectures (all measured via unified pipeline): GPT-2 Small contracts monotonically ( \rho_{\max}=0.96 ), while GPT-2 Medium’s final layer amplifies dramatically ( \rho_{23}=2.05 ). LFM2-2.6B (hybrid) and Mistral-7B show moderate amplification; Qwen3-8B oscillates most. (c) Catastrophe threshold \tau increases with model dimension but is modulated by architecture (Table 2 ). 2 The Living Inference Framework We call our approach “Living Inference” because the approximation adapts continuously: per-matrix error bounds are tracked at every forward pass, enabling the system to detect when approximation quality degrades and adjust accordingly. Each weight matrix in a transformer layer performs a linear map y=Wx . We model these individual matrix multiplications as bounded linear maps f:E\to F between normed spaces, and prove error bounds for three composable approximation stages applied to each map. Crucially, our formal guarantees apply to the linear components —the matrix multiplications themselves—not to the full nonlinear layer (which includes LayerNorm, GELU, and softmax). The composition of errors across nonlinear components is analyzed via Lyapunov stability theory (§ 5.2 ) and validated empirically with zero violations across 14,040+ configurations (§ 5 ). 2.1 Stage 1: Semantic Caching Given a query input x and a cached input x_{c} with \|x-x_{c}\|\leq\delta , we reuse the cached output f(x_{c}) instead of computing f(x) . Theorem 1 (Retrieval Bound) . For any bounded linear map f and inputs x,x_{c} with \|x-x_{c}\|\leq\delta : \|f(x)-f(x_{c})\|\leq\|f\|\cdot\delta This follows directly from the Lipschitz property of bounded linear maps. The bound is tight and provides a direct knob: tighter cache thresholds \delta yield smaller errors. 2.2 Stage 2: Sparse Activation We decompose f=f_{\text{sparse}}+f_{\text{residual}} where f_{\text{sparse}} retains only the top- k % of weights by magnitude. Theorem 2 (Sparse Activation Bound) . If f=f_{\text{sparse}}+f_{\text{residual}} , then: \|f(x)-f_{\text{sparse}}(x)\|\leq\|f_{\text{residual}}\|\cdot\|x\| The error depends on the operator norm of the pruned weights, which decreases monotonically with the retention fraction k . 2.3 Stage 3: Low-Rank Approximation We further approximate f_{\text{sparse}}\approx f_{\text{approx}}+f_{R} using SVD truncation to rank r , where f_{\text{approx}}=U_{r}\Sigma_{r}V_{r}^{\top} and f_{R} captures the discarded singular components. Theorem 3 (Interpolation Bound) . If f_{\text{sparse}}=f_{\text{approx}}+f_{R} , then: \|f_{\text{sparse}}(x)-f_{\text{approx}}(x)\|\leq\|f_{R}\|\cdot\|x\| By the Eckart–Young theorem, the SVD truncation is optimal: \|f_{R}\|=\sigma_{r+1} , the (r+1) -th singular value. 2.4 Composition Theorem The central result combines all three stages into a single bound. Theorem 4 (Living Inference Composition Bound) . Let f=f_{\text{sparse}}+f_{\text{residual}} and f_{\text{sparse}}=f_{\text{approx}}+f_{R} . For query x and cached input x_{c} with \|x-x_{c}\|\leq\delta , the living inference output \hat{y}=f_{\text{approx}}(x_{c})+f_{\text{residual}}(x) satisfies: \|f(x)-\hat{y}\|\leq\underbrace{\|f_{\text{approx}}\|\cdot\delta}_{\text{cache error}}+\underbrace{\|f_{R}\|\cdot\|x\|}_{\text{interpolation}}+\underbrace{\|f_{\text{residual}}\|\cdot\delta}_{\text{sparse-cache interaction}} Each term is independently controllable: \delta (cache threshold), r (SVD rank), and k (sparsity fraction). The composition is additive—no multiplicative interaction between stages—enabling independent tuning. Two additional theorems—efficiency dominance (cache hit reduces total cost) and adaptive monotonicity (tighter \delta never increases error)—are included in the Lean 4 proofs (Appendix A , Theorems 5–6). 3 Formal Verification in Lean 4 All theorems in § 2 are machine-checked in Lean 4 [ 3 ] using the Mathlib library’s normed space infrastructure. 3.1 Proof Architecture We represent neural network layers as bounded linear maps E →L[ℝ] F in Mathlib’s Analysis.NormedSpace module. This provides: • Operator norm \|f\| with the key inequality \|f(x)\|\leq\|f\|\cdot\|x\| ( f.le_opNorm ). • Linearity : f(x-y)=f(x)-f(y) (

Read the complete text