SD 2 : Self-Distilled Sparse Drafters

Mike Lasby, Nish Sinnadurai, Valavan Manohararajah, Sean Lie, Yani Ioannou, Vithursan Thangarasa · 2025

Read SD 2 : Self-Distilled Sparse Drafters by Mike Lasby, Nish Sinnadurai, Valavan Manohararajah, Sean Lie, Yani Ioannou, Vithursan Thangarasa (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08838.txt.

Introduction

Mike Lasby and co-authors speed up large language models with draft models that are both sparse and self distilled. Their 2025 paper works on speculative decoding, where a small drafter guesses tokens that the big model then verifies. Making the drafter cheap without breaking its agreement with the target is the whole game.

Self distillation on the model's own data aligns the drafter, and fine grained weight sparsity trims its arithmetic. Against a seventy billion parameter target, the method raised the mean accepted length well above layer pruned drafters and cut operations sharply against dense ones. Even when drafter and target come from different model families, the gains held. Sparse drafters of 1.5 and 3 billion parameters won on end to end latency.

Catalog record

CREATORMike Lasby, Nish Sinnadurai, Valavan Manohararajah, Sean Lie, Yani Ioannou, Vithursan Thangarasa
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-08838

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SD 2 : Self-Distilled Sparse Drafters Mike Lasby, Nish Sinnadurai, Valavan Manohararajah, Sean Lie, Yani Ioannou, Vithursan Thangarasa Abstract Speculative decoding is a powerful technique for reducing the latency of Large Language Models (LLMs), offering a fault-tolerant framework that enables the use of highly compressed draft models. In this work, we introduce Self-Distilled Sparse Drafters (SD 2 ), a novel methodology that leverages self-data distillation and fine-grained weight sparsity to produce highly efficient and well-aligned draft models. SD 2 systematically enhances draft token acceptance rates while significantly reducing Multiply-Accumulate operations (MACs), even in the Universal Assisted Generation (UAG) setting, where draft and target models originate from different model families. On a Llama-3.1-70B target model, SD 2 provides a 1.59 × \times higher Mean Accepted Length (MAL) compared to layer-pruned draft models and reduces MACs by over 43.87% with a 8.36% reduction in MAL compared to a dense draft models. Our 1.5B and 3B unstructured sparse drafters outperform both dense and layer-pruned models in terms of end-to-end latency improvements; highlighting the potential of sparsity-aware fine-tuning and compression strategies to improve LLM inference efficiency while maintaining alignment with target models. † † Correspondence to [email protected] † † \dagger Work completed while on internship at Cerebras 1 Introduction Large Language Models (LLMs) have proven to have high utility in a wide variety of contexts. However, the causal dependency between preceding and subsequent tokens results in approximately 10 \times higher latency for sequence generation compared to processing an equivalent length sequence in parallel ( Liu et al., 2023 ) . The high computational cost of LLMs has motivated significant research into methods which improve their efficiency, including: quantization ( Gholami et al., 2022 ; Kurtic et al., 2024 ) , pruning ( Ma et al., 2023 ; Gromov et al., 2025 ) , weight sparsity ( Frantar and Alistarh, 2023 ; Sun et al., 2024b ; Yin et al., 2024b ) , activation sparsity ( Mirzadeh et al., 2023 ) , KV-cache compression ( Zhang et al., 2024c ) , distillation ( Kim and Rush, 2016 ; Hsieh et al., 2023 ) , and matrix decomposition ( Hu et al., 2021 ; Liu et al., 2024b ) . However, these methods typically trade improved efficiency for decreased model quality. ( Yin et al., 2024a ; Jaiswal et al., 2024 ) . In contrast, speculative decoding ( Stern et al., 2018 ; Leviathan et al., 2023 ; Chen et al., 2023 ) offers a unique framework to accelerate token generation without sacrificing accuracy. In speculative decoding, a smaller draft model is utilized to auto-regressively generate a sequence of draft tokens which are verified in parallel by a target model. For speculative decoding to be effective, an efficient draft model which is closely aligned with the target model is required. How best to select and/or train a draft model has been the focus of several recent works, see Xia et al. (2024) and Section 4 for more details. Figure 1 : Improvement factor of dense, layer-pruned, and SD 2 unstructured Llama and Qwen models drafting for Llama-3.1-70B-Instruct and Qwen-2.5-72B-Instruct, respectively. SD 2 drafters outperform layer-pruned draft models and dense drafters in the 1.5B and 3B model size categories. (a) UAG Qwen drafting for Llama (b) MACs analysis Figure 2 : ( 2(a) ): SpecBench MAL for SD 2 Qwen-2.5 models drafting for Llama-3.1-70B-Instruct in the UAG setting. These results illustrate the benefits of SD 2 for aligning draft models even across different model families . SD 2 Qwen drafters achieve a higher MAL than their dense counterparts. ( 2(b) ): MAL vs. MACs for layer-pruned and sparse draft models. Particularly notable are the Qwen-2.5 unstructured sparse drafters which approach iso- MAC performance compared to the dense models. Fine-grained sparsity, such as unstructured or 2:4 ( Mishra et al., 2021 ) sparsity, for compressing draft models has yet to be examined. Sparse Neural Networks (SNNs) have significantly reduced connectivity between neurons in adjacent layers compared to dense networks. In unstructured sparsity the active parameters are distributed in an irregular, non-uniform manner throughout the weight matrices which can be challenging to accelerate. To address this, 2:4 sparsity was introduced which offers a hardware-friendly structure in which exactly two out of every four contiguous weights are active. However, the additional constraints of 2:4 sparsity result in lower accuracy on downstream tasks compared to unstructured. Speculative decoding is a uniquely well-suited setting for using sparse draft models since any errors produced during drafting can be gracefully recovered from during the verification step. Other compression strategies such as quantization succeed in reducing memory overhead, but may not reduce latency ( Kurtic et al., 2024 ) . Performing operations with parameters quantized to low bit-width data types requires dequantization to a hardware-native data type, typically 8- or 16-bit types for presently available accelerators. As a result, quantization may lead to increased latency in many circumstances, particularly for relatively small draft models which may not be memory-bound. In contrast, SNNs reduce the total amount of Multiply-Accumulate operations (MACs) required per forward pass which, in an ideal setting, would correspond directly with latency improvements. In practice, accessing non-contiguous sparse parameters and storing the associated sparse structure metadata can lead to significant overhead on Graphics Processing Units (GPUs) ( Hooker, 2020 ) . Achieving real-world acceleration for fine-grained SNNs requires specialized kernels ( Neural Magic, 2021 ; Schultheis and Babbar, 2023 ; Lasby et al., 2023 ; Frantar et al., 2025 ) and hardware such as Cerebras’ wafer-scale engine ( Thangarasa et al., 2024 ; Hall et al., 2023 ; Lie, 2023 ; Agarwalla et al., 2024 ) . Given the practical limitations of fine-grained sparsity, structured pruning — the removal of layers, neurons, or other substructures — of LLMs to obtain performant draft models is a more hardware-friendly alternative. In particular, pruning entire transformer blocks from the decoder of LLMs provides low latency models which retain their quality under moderate compression ratios ( Liu et al., 2023 ; Gromov et al., 2025 ; Men et al., 2024 ; Kim et al., 2024 ; Sun et al., 2025 ) . A natural extension of these works is to leverage layer-pruning techniques to obtain draft models as examined by Thangarasa et al. (2025) . However, it is not clear a priori if the model quality degradation of layer-pruned models can be overcome by their improved latency for use as draft models in the speculative decoding setting. In this work, we propose Self-Distilled Sparse Drafters (SD 2 ) — a novel methodology for obtaining efficient, well-aligned draft models — by utilizing self-data distillation fine-tuning ( Yang et al., 2024b ; Thangarasa et al., 2025 ) and weight sparsity. Specifically, we make the following contributions: • We introduce SD 2 , a novel methodology for obtaining fine-grained sparse draft models; • We demonstrate the superiority of fine-grained sparsity for accelerating speculative decoding and downstream evaluation tasks compared with layer-pruned models; • We showcase the effectiveness of self-data distillation fine-tuning for model alignment, even when aligning with a different model family with Universal Assisted Generation (UAG) ; • When paired with optimized sparse representations, we find that the end-to-end acceleration of speculative decoding with fine-grained sparse draft models is

Read the complete text