An Effective Training Framework for Light-Weight Automatic Speech Recognition Models

Abdul, Alessio, Shah, Mubashir · 2025

Read An Effective Training Framework for Light-Weight Automatic Speech Recognition Models by Abdul, Alessio, Shah, Mubashir (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16991.txt.

Introduction

Abdul Hannan, Alessio Brutti, Shah Nawaz, and Mubashir Noman present a training plan for small speech recognition models. Their paper is from two thousand twenty-five. Large models score well but are too heavy for small devices with tight memory and power. Past ways to shrink them lose much of their skill, or need long training to win it back.

Their two-step plan learns shared representations from one large model, then makes several small ones from it. Each small model trains in few rounds, yet keeps strong scores. On standard speech test sets, training runs about three times as fast. Word error rates also drop by a clear margin next to past routes.

Catalog record

CREATORAbdul, Alessio, Shah, Mubashir
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16991

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Effective Training Framework for Light-Weight Automatic Speech Recognition Models Abdul, Alessio, Shah, Mubashir Abstract Recent advancement in deep learning encouraged developing large automatic speech recognition (ASR) models that achieve promising results while ignoring computational and memory constraints. However, deploying such models on low resource devices is impractical despite of their favorable performance. Existing approaches (pruning, distillation, layer skip etc.) transform the large models into smaller ones at the cost of significant performance degradation or require prolonged training of smaller models for better performance. To address these issues, we introduce an efficacious two-step representation learning based approach capable of producing several small sized models from a single large model ensuring considerably better performance in limited number of epochs. Comprehensive experimentation on ASR benchmarks reveals the efficacy of our approach 1 1 1 Our code is available at: https://github.com/hannabdul/etf4asr , achieving three-fold training speed-up and up to 12.54 % 12.54\% word error rate improvement. Hannan Brutti Nawaz Noman University of TrentoItaly Fondazione Bruno KesslerItaly Johannes Kepler UniversityAustria Mohamed bin Zayed University of Artificial IntelligenceU.A.E. keywords Automatic Speech Recognition, Efficient Training, Feature Representation Learning, Light-weight Model 1 Introduction Accelerated progress in Internet of Things (IoT) [ 1 ] and edge-cloud continuum [ 2 ] demands significant reduction in the neural architecture sizes for smooth operation with minimal computational and memory burden. Several efforts have been directed to create user and device-personalized models with adaptable model sizes while minimizing the latency using different techniques. Contemporary works aims at sparsification of deep models using pruning [ 3 , 4 ] that primarily locate the intermediate representation (or input) that if removed, would minimally affect the model’s output. Others either employ low-precision data and models [ 5 ] or opt for low-rank approximation methods [ 6 , 7 ] , or distill the knowledge from a large teacher model to a student model using a distribution’s divergence-based loss such as Kullback–Leibler divergence, Jensen-shannon divergence, Wasserstein distance, etc [ 8 , 9 , 10 , 11 , 12 , 13 , 14 , 15 ] . However, applying pruning solely leads to substantial performance degradation, directing the researchers to use pruning in conjunction with other approaches [ 4 , 16 , 17 ] to reduce the affect on performance up to some extent. Alternatively, few works utilized conditional computing approaches, such as Early Exit [ 18 ] , Layer dropping [ 19 , 20 ] etc., to reduce the computational load at inference time either by exiting at early stage or skipping some parts of the network. This dynamic behavior comes at the cost of computational flow alteration and addition of new modules, raising the computational and memory requirements. In addition, dynamic dropping of layers leads to considerable reduction in ASR performance. Curating resource-friendly architectures with the ideal performance is the underlying objective of machine learning, and is featuring new paradigms like tiny machine learning (tiny ML), green ML etc. Developing large models with billions of parameters to gain a point over counterparts might not be the right way to improve task-specific performance. Not only these architecture results in tons of carbon dioxide emission and consume a staggering amount of energy until convergence, the level of resource consumption of such large models does not align with these paradigms as well as their deployment on end-user devices is practically unviable. Similarly, compressing large models being undoubtedly effective, suffers from the drawback of inefficient resource utilization in addition to noticeable performance degradation. Moreover, training small and device-conditioned models every time for substantial number of epochs is also unfeasible. The reason is that small models demands prolong training time and large number of epochs to converge with subpar performance level. Therefore, it is desired to build smaller architectures capable of achieving favorable performance as well as requiring reduced computational and memory load. In this regard, we propose an effective approach that exploits the knowledge of large reference model to obtain tiny models capable of performing favorably while requiring low computational resources. To summarize, our contributions are as below. • We propose a two-step based approach to train small ASR models that achieve promising results on ASR task. • The introduced approach enables us to train several small and tiny models requiring less compute resources while providing promising performance compared to equivalent small models. • We perform extensive experiments on two challenging ASR benchmarks demonstrating the effectiveness of proposed approach. Figure 1: Here we illustrate the overall framework of the proposed method. (i) In the feature learning phase, the input utterance is fed to a large ASR model (reference model) to extract the features e_{ref} and b_{ref} . Afterwards, the same input is passed through the smaller model to obtain features e_{LW} and b_{LW} . To learn the knowledge of reference model, MSE loss is used between the outputs ( b_{ref} and b_{LW} ) of the classifiers of the reference and the small model. Symmetric cross-entropy loss is used on the features e_{ref}^{N} and e_{LW}^{M} to align the feature spaces. (ii) After transferring the knowledge to the encoder of light-weight model, CTC decoder is integrated and the model is finetuned to transcribe the input speech. 2 Background Representation Learning using Mask Auto-Encoders: Mask auto-encoders [ 21 , 22 ] are self-supervised frameworks that are utilized to learn rich semantic features from large amount of unlabeled data. Mask auto-encoder masks the large portion of its input X\in\mathbb{R}^{H\times W} , and feeds the unmasked patches to the encoder to obtain encoded features X_{e} . In decoder, learnable features are appended to the encoded features X_{e} which are forwarded to the decoder layers to obtain the reconstructed feature representations X_{r} . Mask auto-encoder utilizes the mean squared error loss (MSE) to measure the reconstruction performance and enable the model to learn meaningful feature representations in self-supervised manner. Knowledge Distillation: Knowledge distillation is a technique utilized to transfer the knowledge of a large or teacher model to a student model. Typically, the teacher model is trained on large set of available data and the knowledge is transferred to the student model by means of distillation loss. Existing works [ 23 , 24 ] have utilized the cross-entropy loss having soft target distributions to distill the knowledge given as: c_{i}=\frac{e^{z_{i}/T}}{\Sigma_{j}e^{z_{j}/T}} (1) where c_{i} is the class probability of ith class, z_{i} is the logit and T is the temperature to control the distillation. This teacher-student architecture is trained in an end-to-end fashion. Given that we have several student networks, it is desired to train each student model using the knowledge distillation framework enabling the student model to perform well on the specific task. In addition to the temperature controlled cross-entropy loss, MSE loss is also utilized to distill knowledge for classification task [ 25 ] . However, MSE is typically used in mask auto-encoder frameworks to learn meaningful representations from huge amount of unlabelled data. Inspired from these, we propose a two-step approach that first learns the semantically rich features from the encoder of large reference model followed by finetuning tiny models for limited number of epochs. 3 Proposed Framework To mitigate the performance limitations and facilitate fast training, we propose a two-step feature learning based framework in Figure 1 . The proposed framework requires:

Read the complete text