Input Conditioned Layer Dropping in Speech Foundation Models
2025
Read Input Conditioned Layer Dropping in Speech Foundation Models (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07954.txt.
Introduction
Abdul Hannan, Daniele Falavigna, and Alessio Brutti present a way to slim speech models on the fly in a 2025 paper. Big pre-trained speech models are accurate but heavy, and the phones or gadgets that run them vary in power from moment to moment. One fix is layer dropping, skipping some of the model's layers to save computing, but older methods pick layers crudely or rebuild the model.
Their approach lets the input itself decide. A small helper network reads the incoming audio and picks which layers to run for that clip. The base model stays untouched. Across four public speech and audio test sets, and two different base models, the method beat random dropping by a wide margin. It also matched or beat early-exit designs, which stop the model partway through.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-07954
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Input Conditioned Layer Dropping in Speech Foundation Models Abstract Curating foundation speech models for edge and IoT settings, where computational resources vary over time, requires dynamic architectures featuring adaptable reduction strategies. One emerging approach is layer dropping ( ℒ 𝒟 \mathcal{LD} ) which skips fraction of the layers of a backbone network during inference to reduce the computational load. This allows transforming static models into dynamic ones. However, existing approaches exhibit limitations either in the mode of selecting layers or by significantly modifying the neural architecture. To this end, we propose input-driven ℒ 𝒟 \mathcal{LD} that employs the network’s input features and a lightweight layer selecting network to determine the optimum combination of processing layers. Extensive experimentation on 4 speech and audio public benchmarks, using two different pre-trained foundation models, demonstrates the effectiveness of our approach, thoroughly outperforming random dropping and producing on-par (or better) results to early exit. Index Terms — dynamic models, layer drop, speech recognition, transformer 1 Introduction Speech foundation models are extensively employed in diverse applications due to their efficient and rich semantic feature representation capability. However, their practicality on low resources / edge devices is limited due to significant computational overhead and enormous memory requirement. To address this issue, efforts have been directed to reduce the resource utilization of large models using different methods, such as pruning [ 1 ] , quantization [ 2 ] , and low-rank decomposition [ 3 , 4 , 5 ] . Although these approaches are fairly effective, they generate fixed architectures that fail to perform in varying resource settings. To this end, split-computing [ 6 , 7 ] and early exit (EE) methods [ 8 , 9 , 10 ] have been introduced allowing the large models to adapt to varying computational resources. However, split-computing may suffer from latency and privacy issues that degrade their effectiveness, while EE methods require both careful selection of exit locations and additional overhead for training auxiliary output layers. Alternatively, inspired from Stochastic depth approach [ 11 ] , Layer Dropping ( \mathcal{LD} ), or Layer Skipping was proposed in [ 12 , 13 ] as a sort of structured pruning technique that drops (or skips) complete layers or modules. Besides improving the training time [ 14 ] , random layer dropping (RD) has been employed in NLP with pretrained foundation models, i.e. BERT, RoBERTa, and XLNet architectures, by dropping pre-defined subnets [ 15 ] . In the Automatic Speech Recognition (ASR) context, [ 9 ] utilized probabilistic \mathcal{LD} to fine-tune WavLM [ 16 ] with a dropping probability of 0.5 . Following that, LDASR [ 17 ] investigated the training and inference behavior of random (or probabilistic) \mathcal{LD} on a conformer-based architecture using a range of dropping probabilities. These studies highlight that \mathcal{LD} provides fair reduction in model size and also induces computational efficiency by skipping layers on random. We hypothesize that RD results in an ineffective combination of selected layers, producing a suboptimal model especially when the available resources are scarce (dropping most of the encoder layers). To this end, we propose an input-driven layer dropping (IDLD) mechanism using a lightweight Layer Selecting (LS) block to transform foundation models in to dynamic architectures suitable for varying resources as well as to exhibit efficient performance even for extreme cases (dropping most of the encoder layers). The LS block that operates in plug-and-play fashion, seamlessly integrates with various foundation models governing which model layers to skip or execute by assigning binary weights to the layers, hence selecting the optimal sub-network inside the foundation model. Although data-driven approaches [ 10 , 12 , 17 , 18 , 19 ] have been used in vision and NLP domains, either their applicability to transformer-based architectures for audio applications, especially ASR, is not investigated or the data-driven approach is unsuitable for extreme cases. In general, our contributions are as follows. Fig. 1 : Illustration of the proposed input-driven layer skipping approach using a pretrained encoder. The output of the feature extractor (X) is forwarded to both Encoder \mathcal{E} and Layer Selector network \mathcal{S} . The latter provides layer scores G\in\mathbb{R}^{N} , for each layer of the encoder. Top-k values from G=\{g^{1},g^{2},..,g^{N}\} are selected to enable corresponding encoder modules while skipping the remaining ones. • We propose input-driven layer dropping (IDLD) mechanism using a plug-and-play LS block, that enables optimal sub-network selection in foundation models for ASR and other audio applications. • We demonstrate efficacy of the proposed mechanism for four downstream audio applications using two foundation models: (i) WavLM for ASR task on two public benchmarks, and (ii) Audio Spectrogram Transformer (AST) [ 20 ] for sound classification, intent classification, and emotion recognition tasks. In addition, we compare threshold-based automatic layer selection with entropy-based early exiting. 2 Background on Dynamic Depth Dynamic architectures have the ability to adapt the network’s computational flow and can be categorized as: (i) dynamic depth [ 12 , 13 , 14 , 15 , 10 , 19 , 17 ] , (ii) dynamic width [ 21 , 18 ] , and (iii) dynamic routing [ 22 , 23 ] . We will restrict the discussion to dynamic depth only as it encapsulates the early exit and layer dropping approaches. Further details on other dynamic networks are available in [ 24 ] . The former offers multiple auxiliary exit branches to limit the processing beyond certain depth. Conversely, layer dropping skips intermediate layers (not necessarily contiguous) adapting the network on the fly without extra classifiers. Next, we discuss the layer dropping approaches in more detail. Layer Dropping/Skipping, initially proposed for ResNet in [ 25 ] , and used in stochastic depth [ 11 ] and Transformers [ 26 ] , has been widely used in computer vision for efficient training. Examples of skipping strategies are: (i) random dropping: (used in NLP applications) where the blocks are dropped randomly while training sub-networks inside the backbone [ 9 , 13 , 17 , 15 , 26 ] , (ii) policy networks: where a separate sub-network predicts the masking value for each block of the backbone network [ 27 , 28 , 29 ] , and (iii) data-driven: where the skipping decision is made on previous block’s output [ 12 , 18 , 30 , 31 ] . Recently, in vision domain, researchers put forward different approaches leveraging ResNet architecture to skip the model’s layers. [ 27 ] opted for reinforcement learning based skipping decisions whereas [ 12 ] used a hybrid of reinforcement and supervised learning to select the dropping layers on the basis of previous layer’s output. [ 29 ] proposed to use a decision module for each layer separately, thus preventing to use effectively the available resources. Furthermore, [ 28 ] suggested to process the input separately from the backbone network to compute which layers should be dropped/executed. Aforementioned, these methods are yet to be investigated for pre-trained transformer architectures without architectural alteration. For what concerns the ASR domain, work has been done using RD [ 9 , 17 ] and progressive layer skipping with a hybrid training methodology [ 32 ] , achieving reduced computational complexity coupled with fast convergence. In case of data-driven methods, [ 10 ] employed EE, and