BitSkip: An Empirical Analysis of Quantization and Early Exit Composition

Ramshankar Bhuvaneswaran, Handan Liu · 2025

Read BitSkip: An Empirical Analysis of Quantization and Early Exit Composition by Ramshankar Bhuvaneswaran, Handan Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-23766.txt.

Introduction

Ramshankar Bhuvaneswaran and a co-author study what happens when two popular tricks for shrinking language models are combined. One is quantization, which stores weights in fewer bits. The other is early exit, which lets easy inputs leave the network midway. Each trick helps alone, but their combined behavior was poorly understood. Their framework, BitSkip, varies both systematically to map the interactions.

The results surprised them. A plain 8-bit model with no Hadamard transform beat its fancier 4-bit and Hadamard-enhanced siblings, and nearly matched the full-precision baseline on perplexity. Adding Hadamard transforms wrecked training stability, degrading quality catastrophically even at 8 bits. The simple recipe also exited early gracefully: stopping at layer eighteen bought a speed gain near a third for only about four percent quality loss. The moral is that stacking optimizations can backfire, and the plainest combination won.

Catalog record

CREATORRamshankar Bhuvaneswaran, Handan Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-23766

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BitSkip: An Empirical Analysis of Quantization and Early Exit Composition Ramshankar Bhuvaneswaran, Handan Liu Abstract The pursuit of efficient Large Language Models (LLMs) has led to increasingly complex techniques like extreme quantization and dynamic routing. While individual benefits of these methods are well-documented, their compositional effects remain poorly understood. This paper introduces BitSkip, a hybrid architectural framework for systematically exploring these interactions. Counter-intuitively, our findings reveal that a simple 8-bit quantized model without Hadamard transform (BitSkip-V1) not only outperforms its more complex 4-bit and Hadamard-enhanced counterparts but also competes the full-precision baseline in quality (perplexity of 1.13 vs 1.19). The introduction of Hadamard transforms, even at 8-bit precision, catastrophically degraded performance by over 37,000%, tracing fundamental training instability. Our BitSkip-V1 recipe demonstrates superior early-exit characteristics, with layer 18 providing optimal 32.5% speed gain for minimal 4% quality loss. Keywords Large Language Models \cdot Quantization \cdot Early Exit \cdot Dynamic Neural Networks \cdot Model Efficiency 1 Introduction The inflation of Large Language Models has reached a point where physical hardware constraints like memory bandwidth, computational throughput, and power consumption increasingly dictate architectural possibilities [ 1 ] . This reality has led researchers to shift focus into efficiency techniques, with quantization and dynamic neural networks emerging as two prominent approaches. Quantization methods have evolved from post-training compression [ 7 , 9 ] to extreme co-designed architectures like BitNet [ 13 , 10 ] that train models from scratch with 1-bit weights. Simultaneously, dynamic networks like LayerSkip [ 5 ] have demonstrated that adaptive computation based on input complexity can provide substantial speedups. However, a critical gap persists: these techniques have been developed and evaluated in isolation. The fundamental question of how quantization and dynamic routing interact compositionally remains unexplored. Current literature lacks empirical models to predict whether combining efficiency techniques yields multiplicative benefits or destructive interference. This paper introduces BitSkip, a unified framework for investigating these interactions through systematic ablation across three architectural variants. Our key findings reveal: (1) Our BitSkip-V1 (8-bit, no Hadamard) achieves superior quality (PPL 1.13 vs 1.19) compared to full-precision baseline. (2) Hadamard transforms can be destructive, degrading performance by 37,757% even at 8-bit precision. (3) Early exit viability depends fundamentally on base architecture stability. Our work provides the first comprehensive empirical analysis of quantization-early exit composition, establishing clear design principles for next-generation efficient LLMs. 2 Related Work 2.1 Quantization Techniques Quantization reduces the numerical precision of model parameters to improve efficiency. Post-training quantization methods like GPTQ [ 7 ] and AWQ [ 9 ] enable rapid compression but face accuracy degradation from activation outliers. On the other hand, Quantization-aware training (QAT) as surveyed by Nagel et al. 2021 , incorporates precision constraints during training, often employing straight-through estimators [ 3 ] for gradient approximation. This approach consistently outperforms PTQ methods, particularly at aggressive precision levels, though it requires additional computational resources for fine-tuning. This field has recently witnessed breakthroughs by BitNet family [ 13 , 10 ] represents a breakthrough in extreme quantization, demonstrating that transformer architectures can be trained from scratch with 1-bit weights while maintaining competitive performance. Bitnet-v2 [ 14 ] further introduced Hadamard transformations to enable stable 4-bit activations. 2.2 Dynamic Neural Networks Dynamic networks adjust computational requirements based on input complexity, recognizing that not all inputs require equal processing. The theoretical foundation was established by [ 2 ] with methods for gradient propagation through stochastic neurons. LayerSkip [ 5 ] implements this through layer dropout during training with random layer dropout, teaching it to be robust when certain layers are skipped. This creates a network that can function effectively even when using only a subset of its layers. and self-speculative decoding, this technique uses tokens generated by early exits as “draft” predictions. These drafts are then verified in parallel by the full model, allowing for both speed and accuracy. This LayerSkip laid the groundwork for networks that could make discrete decisions about which computational paths to follow. Building on these ideas, early research explored adaptive depth networks and early exiting mechanisms. Recent work like DEL [ 6 ] has further refined dynamic exit layer selection. 2.3 The Composition Gap Despite progress in both domains, the compositional effects of combining quantization with dynamic routing remain unexplored. [ 1 ] formalized the challenge of “algorithmic speedup” composition but noted the lack of empirical models for predicting interaction effects. Our work directly addresses this gap through systematic experimentation. 3 The BitSkip Framework BitSkip provides a unified framework for investigating quantization-early exit interactions through three architectural variants: 3.1 Core Architecture All BitSkip variants integrate BitLinear layers with LayerSkip’s routing mechanism. The forward pass for layer l is defined as: h_{l}=\text{LayerSkip}(\text{BitLinear}(h_{l-1},W_{l}),p_{d}(l)) (1) where \text{BitLinear}(\cdot) implements a precision-constrained linear transformation utilizing ternarized weight matrices W_{l}\in\{-1,0,+1\}^{d\times d} . This ternary quantization follows a deterministic rounding scheme during forward propagation while maintaining high-precision gradients during backward passes through straight-through estimators [ 3 ] for stable training. The ternary constraint in Equation 1 fundamentally alters the computational characteristics, replacing floating-point multiply-accumulate operations. The \text{LayerSkip}(\cdot,p_{d}) operator in Equation 1 implements stochastic depth during training and conditional execution during inference. This dynamic routing mechanism executes the internal transformation with probability 1-p_{d}(l) , where p_{d}(l) follows a quadratic scheduling function across network depth. During forward passes where the layer is skipped, the operator functions as an identity mapping, passing the input h_{l-1} unchanged to the subsequent layer. This design enables the network to learn robust representations that remain effective even when arbitrary subsets of layers are bypassed. The quadratic dropout schedule represents a key innovation over prior exponential schedules, defined as: p_{d}(l)=c\cdot(l/L)^{2} (2) where c is a scaling constant calibrated to achieve target early exit rates, l is the current layer index, and L is the total layer count. This quadratic formulation in Equation 2 provides smoother gradient flow across layers and improved training stability compared to more aggressive exponential schedules. For architectures employing Hadamard transformation (BitSkip-V2 and V3), we implement the transform as: \text{Hadamard}(X)=H_{m}\cdot X (3) where H_{m} is a 2^{m}\times 2^{m} Hadamard matrix defined recursively as: H_{m}=\frac{1}{\sqrt{2}}\begin{bmatrix}H_{m-1}&H_{m-1}\\ H_{m-1}&-H_{m-1}\end{bmatrix},\quad H_{0}=(1) (4) and X\in\mathbb{R}^{n} with n=2^{m} . We employ fast Hadamard transform algorithms to perform this multiplication with O(n\log n) complexity instead of the naive O(n^{2}) , making the transformation computationally feasible for large-scale language models while preserving its benefits of activation decorrelation and outlier mitigation. 3.2 Architectural Variants BitSkip-V1 (8-bit, No Hadamard): This variant employs 8-bit weight quantization without Hadamard transformation, representing the most straightforward composition of quantization and early exit mechanisms which is similar architecture as that were pioneered and scaled in the BitNet b1.58 work [ 11 ] . The 8-bit precision provides a balanced trade-off between computational efficiency and representation fidelity, maintaining sufficient numerical precision for stable gradient propagation while significantly reducing memory bandwidth requirements. BitSkip-V2 (4-bit + Hadamard): Building upon the BitNet v2 foundation [ 14 ] , this variant incorporates aggressive 4-bit weight quantization coupled with Hadamard transformation for activation preconditioning (Equations 3 and 4 ). The Hadamard transform serves to decorrelate activation dimensions and reduce the impact of outlier values, theoretically enabling more stable low-precision quantization. This variant represents the most extreme quantization approach, targeting maximum compression while relying on mathematical transformations to maintain representational capacity. BitSkip-V3 (8-bit

Read the complete text