Distillation-based Layer Dropping (DLD): Effective End-to-end Framework for Dynamic Speech Networks

Abdul Hannan, Daniele Falavigna, Shah Nawaz, Mubashir Noman, Markus Schedl, Alessio Brutti · 2026

Read Distillation-based Layer Dropping (DLD): Effective End-to-end Framework for Dynamic Speech Networks by Abdul Hannan, Daniele Falavigna, Shah Nawaz, Mubashir Noman, Markus Schedl, Alessio Brutti (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16117.txt.

Introduction

Abdul Hannan and coauthors make speech nets that can shed layers to fit small gear. Their 2026 preprint deals with edge gear with tight and shifting means. A fixed net can be too big for weak gear. Layer drop turns a fixed net into one that can skip parts. Skips cut cost but can hurt speech scores if done with no care.

The team adds a teach step to the drop plan. A full net teaches the lean net as it learns to skip. That teach step aims to keep scores high at low cost. The net can then shift its size as gear means shift. The work targets live speech tools on phones and like gear.

Catalog record

CREATORAbdul Hannan, Daniele Falavigna, Shah Nawaz, Mubashir Noman, Markus Schedl, Alessio Brutti
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE19 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16117

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Distillation-based Layer Dropping (DLD): Effective End-to-end Framework for Dynamic Speech Networks Abdul Hannan, Daniele Falavigna, Shah Nawaz, Mubashir Noman, Markus Schedl, Alessio Brutti Abstract Edge devices operate in constrained and varying resource settings, requiring dynamic architectures that can adapt to limitations of the available resources. To meet such demands, layer dropping ( ℒ ​ 𝒟 \mathcal{LD} ) approach is typically used to transform static models into dynamic ones by skipping parts of the network along with reducing overall computational complexity. However, existing ℒ ​ 𝒟 \mathcal{LD} methods greatly impact the dynamic model’s performance for low and high dropping cases, deteriorating the performance-computation trade-off. To this end, we propose a distillation-based layer dropping (DLD) framework that effectively combines the capabilities of knowledge distillation and ℒ ​ 𝒟 \mathcal{LD} in an end-to-end fashion, thereby achieving state-of-the-art performance for dynamic speech networks. Comprehensive experimentation utilizing well-known speech recognition methods, including conformer and WavLM, on three public benchmarks demonstrates the effectiveness of our framework, reducing the word error rate by 9.32 % 9.32\% and 2.25 % 2.25\% for high and no dropping cases with 33.3 % 33.3\% reduction in training time. † † address: 1 University of Trento, Italy, 2 Fondazione Bruno Kessler, Italy, 3 Johannes Kepler University Linz, Austria, 4 MBZUAI U.A.E., 5 Linz Institute of Technology, Austria 1 Introduction Speech foundation models offer promising performance on various downstream tasks such as automatic speech recognition (ASR), spoken language understanding, speaker identification etc., due to their rich semantic representation capability. This favorable performance is achieved at the cost of enormous computational complexity limiting their deployment on the resource-constrained devices with resource variability. Conventional techniques like pruning [ 15 , 27 ] , quantization [ 13 ] and knowledge distillation [ 11 ] renders compression with relatively good performance-computation trade-off, however, these methods do not suffice the requirements of a dynamic architecture that is preferred for varying-resource devices. To achieve dynamic architectures that are well-suited for such scenarios, techniques like data-offloading: presents privacy and latency issues [ 14 , 17 ] , early exit: involves delicate design choices for inserting exit branches [ 20 ] , and especially layer dropping (or skipping) are utilized enabling the architecture to perform efficiently in such dynamic environment. Layer dropping \mathcal{(LD)} , inspired from stochastic depth [ 12 ] in vision domain, has been increasingly utilized to transform static architectures into dynamic ones [ 12 , 4 , 21 ] . \mathcal{LD} , also referred as structured pruning, executes some parts of the network and skips the rest, providing regularization during training phase and improving execution time, and can be categorized as: (i) random dropping (RD): where each module can be dropped depending on probability [ 4 , 26 , 25 , 7 ] , and (ii): data-driven : where input of the model constitutes the dropping blueprint [ 21 , 3 , 19 , 9 , 5 ] . Zhang and He [ 26 ] proposed a gating scheme for RD to skip different subnets of pre-trained networks using BERT, RoBERTa, and XLNet architectures. In context of ASR domain, recent work [ 8 ] presented an effective two-step framework to create multiple small sized static models, however, the training recipe requires enhancement as well as the resulting models are not suitable for varying resource environments. Another study [ 25 ] compared the performance of different approaches, i.e., early exit, input downsampling, and RD with a dropping probability p_{d}=0.5 , by fine-tuning a pre-trained network on a subset of dataset. However, they omit the evaluation for extreme dropping scenarios. LDASR [ 7 ] investigated the inference time behavior of conformer architecture trained using RD with p_{d}=\{0.2,0.5,0.8\} , and revealed that p_{d}=0.5 provides optimum performance-computation trade-off. However, their approach offers substantial performance degradation for high dropping values as well as struggles to exploit the computational capacity of the complete model in case of no (or low) dropping. Similarly, other works attempts to achieve a dynamic network using learnable masking [ 5 ] , and data-driven methods [ 19 , 22 ] , and either omit evaluation or offer significant performance drop for high dropping values. Figure 1: Illustration of proposed DLD framework. \mathcal{M}_{\text{ref}} uses all encoder layers to supervise the embeddings of \mathcal{M}_{\text{DS}} . Besides, knowledge distillation (KD) has been extensively utilized to achieve model compression for ASR downstream task, resulting in compressed static models. TutorNet [ 24 ] performs KD from multiple locations of teacher model, whereas others [ 1 , 23 ] utilized several teacher networks to supervise the student model. In this regard, inspired from two-step framework of [ 8 ] , this work aims to address the aforementioned issues of dynamic architectures using knowledge distillation in conjunction with random dropping and is the first of its kind to employ KD with dynamic architectures for ASR to the best of our knowledge. To summarize, our contributions are: 1. We propose an end-to-end DLD framework that distills expert knowledge of large model to the dynamic student model, featuring promising performance. 2. The introduced approach resolves the performance degradation issue for no dropping, in addition to improving overall performance for all model sizes. 3. We perform comprehensive experimentation on two publicly available ASR benchmarks using conformer and WavLM architectures, demonstrating the superiority of proposed framework. 2 Methodology To enhance the inference time efficacy of a dynamic model and mitigate performance degradation issues, we propose an end-to-end DLD 1 1 1 Our code is available online https://github.com/hannabdul/DLD4ASR framework that aligns the distribution of latent embeddings of a dynamic student model \mathcal{M}_{\text{DS}} to a reference model \mathcal{M}_{\text{ref}} . Architecture and Data Flow: \mathcal{M}_{\text{ref}} consists of a feature extractor \mathcal{F}_{\text{ref}} , encoder \mathcal{E} with N_{\text{ref}} transformer/conformer modules, and a linear projector whose output undergoes a softmax operation to get latent embedding l_{\text{ref}} . Similarly, \mathcal{M}_{\text{DS}} comprises of a feature extractor \mathcal{F}_{\text{DS}} , encoder \mathcal{E}_{\text{DS}} with a total of N_{\text{DS}} transformer/conformer modules from which n_{\text{DS}} are used per iteration, and a linear projector followed by a softmax, providing latent embedding l_{\text{DS}} . The \mathcal{E}_{\text{DS}} is equipped with a gated mechanism that allows to process or skip the i -th encoder module using a gate g^{i} . The gate g^{i}\in\{0,1\} follows the bernoulli distribution (BD) with a probability of 0.5. y^{i+1}=y^{i}+g^{i}\cdot f^{i}(y^{i}) (1) where y^{i} and y^{i+1} are the input and output of i -th encoder block, and f^{i}(.) is the transformation applied by the i -th encoder. As g^{i} follows the bernoulli distribution, n_{\text{DS}} also becomes a bernoulli random variable whose value can be determined as n_{\text{DS}}=\sum_{i=1}^{N_{\text{DS}}}g^{i} for each iteration during training period. At inference, the value of n_{\text{DS}} is defined enabling us to evaluate the model’s performance to different encoder depths, i.e. n_{\text{DS}}=2,4,\dots,N_{\text{DS}} . For a dataset \mathcal{D}\in\{(a^{j},t^{j})\}_{j=1}^{J} comprising of J audio-transcription pairs, \mathcal{M}_{\text{ref}} produces a latent space representation l_{ref}^{j}|\mathcal{E}_{N_{\text{ref}}} for the j -th input using N_{\text{ref}} encoder modules. In contrast, \mathcal{M}_{\text{DS}} utilizes n_{\text{DS}} number of encoder modules for each iteration to produce the latent space representation l_{\text{DS}}^{j}|\mathcal{E}_{n_{\text{DS}}} where 1\leq n_{\text{DS}}\leq N_{\text{DS}} . Objective Function: To align the distributions of reference model’s embedding l_{\text{ref}}^{j}|\mathcal{E}_{N_{\text{ref}}} and dynamic student model’s embedding l_{\text{DS}}^{j}|\mathcal{E}_{n_{\text{DS}}} , we employ Kullback–Leibler divergence ( \mathcal{L}_{\texttt{KLD}} ) loss which minimizes the statistical distance between both embeddings. \mathcal{L}_{\texttt{KLD}}=\text{min}\sum_{j=i}^{J}\text{D}_{\texttt{KL}}\bigl(l_{\text{ref}}^{j}\>||\>\>l_{\text{DS}}^{j}\bigr) (2) In addition, the

Read the complete text