Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity
Di Zhang, Xun Wu, Shaohan Huang, Yudong Wang, Hanyong Shao, Yingbo Hao, Zewen Chi, Li Dong, Ting Song, Yan Xia, Zhifang Sui, Furu Wei, [0.4em] Microsoft Research · 2026
Read Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity by Di Zhang, Xun Wu, Shaohan Huang, Yudong Wang, Hanyong Shao, Yingbo Hao, Zewen Chi, Li Dong, Ting Song, Yan Xia, Zhifang Sui, Furu Wei, [0.4em] Microsoft Research (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05168.txt.
Introduction
Di Zhang and team show that 1.58-bit BitNet models tolerate semi-ordered thinness better than full-precision models. Their Sparse-BitNet method applies low-bit quantization and dynamic N:M sparsification as one while keeping training stable for the first time. Across model sizes and training schedules, the low-bit models lose less accuracy at equal thinness and survive higher ordered thinness before collapsing. A custom sparse tensor core turns the mix into large training and inference speedups, pointing to a hopeful route for lean language models.
Catalog record
| CREATOR | Di Zhang, Xun Wu, Shaohan Huang, Yudong Wang, Hanyong Shao, Yingbo Hao, Zewen Chi, Li Dong, Ting Song, Yan Xia, Zhifang Sui, Furu Wei, [0.4em] Microsoft Research |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38101 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.05168
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Sparse-BitNet: 1.58-bit LLMs are Naturally Friendly to Semi-Structured Sparsity Di Zhang, Xun Wu, Shaohan Huang, Yudong Wang, Hanyong Shao, Yingbo Hao, Zewen Chi, Li Dong, Ting Song, Yan Xia, Zhifang Sui, Furu Wei, [0.4em] Microsoft Research Abstract Semi-structured N:M sparsity and low-bit quantization (e.g., 1.58-bit BitNet) are two promising approaches for improving the efficiency of large language models (LLMs), yet they have largely been studied in isolation. In this work, we investigate their interaction and show that 1.58-bit BitNet is naturally more compatible with N:M sparsity than full-precision models . To study this effect, we propose Sparse-BitNet , a unified framework that jointly applies 1.58-bit quantization and dynamic N:M sparsification while ensuring stable training for the first time. Across multiple model scales and training regimes (sparse pretraining and dense-to-sparse schedules), 1.58-bit BitNet consistently exhibits smaller performance degradation than full-precision baselines at the same sparsity levels and can tolerate higher structured sparsity before accuracy collapse. Moreover, using our custom sparse tensor core, Sparse-BitNet achieves substantial speedups in both training and inference, reaching up to 1.30 × \times . These results highlight that combining extremely low-bit quantization with semi-structured N:M sparsity is a promising direction for efficient LLMs. Code available at https://github.com/AAzdi/Sparse-BitNet 1 Introduction Large Language Models (LLMs) have demonstrated remarkable performance across a wide range of tasks ( 27 ; 28 ; 19 ; 38 ) . However, their rapidly increasing scale leads to substantial training and inference costs ( 19 ; 38 ) , making efficiency a central challenge in modern LLMs research. Among existing approaches, Quantization ( 18 ; 11 ; 7 ) and Sparsity ( 15 ; 33 ; 14 ) have emerged as two widely studied and effective strategies for improving LLM efficiency. Semi-structured N:M sparsity, particularly the widely supported 2:4 pattern ( 10 ; 14 ; 33 ) , has attracted increasing attention for its ability to accelerate matrix multiplication on NVIDIA Sparse Tensor Cores ( 26 ) , which require that at most 2 out of every 4 weights are non-zero to exploit sparsity for hardware acceleration. Nevertheless, existing works ( 10 ; 13 ; 16 ) have primarily applied semi-structured sparsity to full-precision LLMs. Under strict N:M constraints, these full-precision models often suffer rapid accuracy degradation, making it challenging to achieve both high sparsity and high performance simultaneously. Figure 1 : Intrinsic Sparsity in 1.58-bit BitNet. We present the aggregated weight statistics averaged across all linear layers of the pre-trained 1.58-bit BitNet (2B) model ( 22 ) . (A) The distribution of normalized latent weights exhibits a distinct quantization-valley structure, where the majority of values fall within the [-0.5, 0.5] rounding interval. (B) Consequently, the quantized discrete states are dominated by zeros (approx. 42.3%), confirming that BitNet naturally converges to a highly sparse representation without explicit pruning . Figure 2 : Comparison of N:M sparsity friendliness between 1.58-bit BitNet and full-precision models. Normalized PPL increase relative to each method’s dense (8:8) counterpart. The dashed line marks a 10% degradation threshold. At 2:4 (50% sparsity; same ratio as 4:8), BF16 exceeds the threshold (+18.8%) while BitNet remains below it (+5.7%), indicating 1.58-bit BitNet is more sparsity-friendly than full-precision models. In parallel, extremely low-bit quantization has emerged as an alternative pathway toward LLM efficiency. In particular, the 1.58-bit BitNet ( 34 ; 36 ; 21 ) quantizes weights into a ternary set \{-1,0,1\} and achieves performance competitive with full-precision baselines at scale ( 21 ) . As shown in Figure 1 , the weights of a pretrained 1.58-bit BitNet exhibit a distinctive quantization-valley structure, with a high fraction of zero-valued ternary weights (approximately 42%), naturally forming a sparse representation without explicit pruning. Although these zeros are unstructured and do not directly enable N:M sparse kernel acceleration, they reveal a weight-magnitude geometry that is inherently more compatible with magnitude-based N:M sparsity selection. This observation highlights an important yet underexplored gap in the literature. While prior work has extensively studied N:M sparsity in full-precision models and low-bit quantization in isolation, the interaction between extremely low-bit quantization and semi-structured sparsity remains largely unexplored. This gap motivates the following research question: Under the same N:M sparsity constraints, is 1.58-bit BitNet more sparsity-friendly than full-precision models? To address this question, we first develop a unified Sparse-BitNet training framework (§ 2 ) that jointly enforces 1.58-bit BitNet weight quantization and N:M semi-structured sparsity constraints while ensuring stable training of LLMs. Using this framework, we compare the sparsity-friendliness of 1.58-bit BitNet and BF16 models under two settings: (1) from-scratch pretraining with N:M sparsity constraints, and (2) dense-to-sparse training schedules that switch from dense to N:M sparse training at different stages. Experiments on the Qwen-2.5 model family ( 32 ) across multiple scales (0.5B–3B) show that 1.58-bit BitNet consistently incurs smaller performance degradation than BF16 under identical sparsity constraints (see in Figure 2 ). In addition, we measure end-to-end inference throughput on NVIDIA GPUs using our in-house 6:8 sparse operator across varying sequence lengths and batch sizes, and find that combining 1.58-bit BitNet with N:M sparsity yields substantial speedups (reaching up to 1.30 \times ) in both training and inference. Our main contributions are summarized as follows: • We first systematically investigate and show that extremely low-bit quantization (e.g., 1.58-bit BitNet) is inherently more compatible with semi-structured N:M sparsity than full-precision (e.g., BF16) models, exhibiting smaller accuracy degradation under identical sparsity constraints. • We propose a Sparse-BitNet training framework that jointly integrates 1.58-bit quantization and N:M sparsity to improves training stability and robustness (§ 2 ). • Extensive experiments demonstrate that Sparse-BitNet achieves better accuracy–efficiency trade-offs than sparse full-precision baselines (see in Figure 2 ), with favorable end-to-end speedups (§ 3 ). 2 Sparse-BitNet In this section, we present Sparse-BitNet, a method that integrates semi-structured sparsity into the ternary quantization training landscape. We first review the foundational components—BitNet b1.58 and N\!:\!M sparsity—and then detail our proposed architecture, training strategy, and the interaction between these two compression dimensions. 2.1 Preliminaries 1.58-bit BitNet. 1.58-bit BitNet 22 is an evolution of the BitNet architecture 34 that constrains weights to a ternary set \mathcal{T}=\{-1,0,+1\} , resulting in a theoretical information density of \log_{2}3\approx 1.58 bits per parameter. The fundamental building block is the BitLinear layer. For a weight matrix \mathbf{W}\in\mathbb{R}^{d_{\text{out}}\times d_{\text{in}}} , the ternary weights \mathbf{W}_{q} are derived by scaling the latent full-precision weights by their average absolute magnitude: \mathbf{W}_{q}=\mathrm{RoundClip}\left(\frac{\mathbf{W}}{\gamma+\epsilon},-1,+1\right), (1) where \gamma=\frac{1}{d_{\text{out}}d_{\text{in}}}\|\mathbf{W}\|_{1} serves as the scaling factor and \epsilon is a small constant for numerical stability. Activations are quantized to 8-bit integers using absmax scaling: \tilde{\mathbf{x}}=\mathrm{Quant}(\mathbf{x})=\mathrm{Clip}(\mathbf{x}\cdot\frac{127}{\max(|\mathbf{x}|)+\epsilon},-128,127) . The forward pass is then computed as \mathbf{y}\approx\gamma\cdot\mathrm{matmul}(\mathbf{W}_{q},\tilde{\mathbf{x}}) , significantly reducing the computational overhead by replacing floating-point multiplications with integer additions and scaling. Semi-Structured N:M Sparsity. Semi-structured sparsity enforces a fine-grained pattern where at most N elements are non-zero out of every M consecutive weights 41 . This format retains hardware efficiency (e.g., via Sparse Tensor Cores) while offering flexibility over coarse pruning. For a weight matrix \mathbf{W} , we define a binary mask \mathbf{M}\in\{0,1\}^{d_{\text{out}}\times d_{\text{in}}} such that for every group of M elements (typically along the input dimension), \|\mathbf{M}_{\text{group}}\|_{0}=N . The sparse computation is performed as \mathbf{y}=(\mathbf{W}\odot\mathbf{M})\mathbf{x} . In this work, we focus on the 6\!:\!8 pattern ( 25\% sparsity) as a balanced trade-off between compression and accuracy for