Efficient Speech Translation through Model Compression and Knowledge Distillation

Yasmin Moslem · 2025

Read Efficient Speech Translation through Model Compression and Knowledge Distillation by Yasmin Moslem (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20237.txt.

Introduction

Moslem reports shrinking a speech translation model for the IWSLT twenty twenty-five shared task in twenty twenty-five. Large audio language models translate speech well but demand heavy computing. Her system entries target the model compression track. The base model was Qwen2-Audio, translating speech into German and Chinese.

Three methods were combined to slim the model. Unimportant layers were pruned step by step. Low rank adapters with four bit quantization cut memory further. Knowledge distillation taught the small model from the large one. The compressed models halved in size and parameter count. They kept ninety-seven to one hundred percent of the original translation quality. The work shows practical routes to deploy speech translation on smaller hardware.

Catalog record

CREATORYasmin Moslem
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20237

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Efficient Speech Translation through Model Compression and Knowledge Distillation Yasmin Moslem Abstract Efficient deployment of large audio-language models for speech translation remains challenging due to their significant computational requirements. In this paper, we address this challenge through our system submissions to the “Model Compression” track at the International Conference on Spoken Language Translation (IWSLT 2025). We experiment with a combination of approaches including iterative layer pruning based on layer importance evaluation, low-rank adaptation with 4-bit quantization (QLoRA), and knowledge distillation. In our experiments, we use Qwen2-Audio-7B-Instruct for speech translation into German and Chinese. Our pruned (student) models achieve up to a 50% reduction in both model parameters and storage footprint, while retaining 97-100% of the translation quality of the in-domain (teacher) models. 1 Introduction Multimodal foundation models have shown powerful capabilities in different tasks, including speech translation. However, these models are often large and computationally intensive, making them impractical to use in real-world settings with limited resources. To enhance the efficiency of these models, researchers have been investigating diverse approaches to model compression that aim to reduce the computational requirements while retaining performance ( Gandhi et al., 2023 ; Peng et al., 2023b ; Peng et al., 2023a ; Treviso et al., 2023 ; Wang et al., 2023 ) . Qwen2-Audio ( Chu et al., 2023 ; Chu et al., 2024 ) is a state-of-the-art foundation model that accepts various audio signal inputs and performs audio analysis or direct textual responses to speech instructions. In the IWSLT 2025’s Model Compression track ( Abdulmumin et al., 2025 ) , the organizers required that all submissions must be derived from the Qwen2-Audio model. The official languages of the task are English-to-German (EN-DE) and English-to-Chinese (EN-ZH). We experimented with various approaches including efficient fine-tuning using quantized low-rank adapters with 4-bit quantization (QLoRA) Hu et al. (2021) ; Dettmers et al. (2023) , iterative layer pruning based on layer importance evaluation ( Peer et al., 2022 ; Gandhi et al., 2023 ; Sajjad et al., 2023 ) , and sequence-level knowledge distillation ( Kim and Rush, 2016 ; Crego and Senellart, 2016 ; Jooste et al., 2022 ; Gandhi et al., 2023 ) . Our experiments are mainly based on Qwen2-Audio-7B-Instruct . 1 1 1 https://hf.co/Qwen/Qwen2-Audio-7B-Instruct While Section 3 elaborates on our experiments, we can summarize our two submissions for German and Chinese as follows: • Setup 1: This model is the outcome of fine-tuning Qwen2-Audio-7B-Instruct in two stages: (a) full fine-tuning with the ACL 60/60 dataset, and (b) QLoRA fine-tuning with 4-bit quantization using the ACL 60/60 dataset augmented with data knowledge distillation from the fully fine-tuned model. This process has achieved 40% compression in terms of both model parameters and storage size. We discuss the details of this model in Section 3.2 . • Setup 2: This model is a pruned version of Qwen2-Audio-7B-Instruct , and was created in multiple stages: (a) full fine-tuning of the baseline model with the ACL 60/60 dataset, (b) layer pruning of the decoder into 24 layers, while all 32 encoder layers were kept intact, (c) full fine-tuning of the pruned model, and (d) QLoRA fine-tuning with the ACL 60/60 dataset augmented with data knowledge distillation from the fully fine-tuned model and a portion of the CoVoST2 dataset to restore the quality of the teacher model. This process has achieved 50% compression in terms of both model parameters and storage size. We discuss the details of this model in Section 3.3 . EN-DE EN-ZH Params Storage Model BLEU chrF++ COMET BLEU chrF COMET (B) (GB) Baseline 22.96 49.88 8.61 38.73 32.53 17.38 8.40 16.79 + Full Fine-tuning 39.28 65.27 56.32 58.54 52.54 65.97 + QLoRA Fine-tuning 41.52 66.25 53.47 57.65 51.08 65.77 4.95 9.64 + Knowledge Distillation 43.25 68.02 59.36 59.60 53.94 68.42 Table 1: Evaluation of the experiment that employs QLoRA with 4-bit quantization and augments the authentic data with knowledge distillation. This approach reduces the model size by more than 40% in terms of both the number of parameters (params) and storage footprint, while achieving the best translation performance for both English-to-German (EN-DE) and English-to-Chinese (EN-ZH) language pairs. 2 Data 2.1 In-Domain Data The ACL 60/60 dataset 2 2 2 https://hf.co/datasets/ymoslem/acl-6060 ( Salesky et al., 2023 ) is used in all of our experiments as the in-domain data. ACL 60/60 contains multilingual translation of ACL 2022 technical presentations into 10 target languages. The dataset consists of two splits, “dev” and “eval”, which together comprise 884 utterances. We merged the two splits, and randomly sampled 100 utterances for testing, which left us with 784 utterances for training. For test data sampling, we used the train_test_split method from the datasets 3 3 3 https://github.com/huggingface/datasets library ( Lhoest et al., 2021 ) , setting the random seed option to zero. The ACL 60/60 dataset was required for “constrained” submissions to the IWSLT’s “Model Compression” track. 2.2 Out-of-Domain Data After layer pruning of the Qwen2-Audio-7B-Instruct model (cf. Section 3 ), we needed to use more training data to restore the translation quality of the unpruned model. In addition to knowledge distillation data from the teacher model, we used a portion of the CoVoST2 dataset ( Wang et al., 2021 ) (cf. Section 3.3 ). CoVoST2 is a large-scale multilingual speech-to-text translation corpus, covering translations from English into 15 languages, including German and Chinese. EN-DE EN-ZH Params Storage Model Data BLEU chrF++ COMET BLEU chrF COMET (B) (GB) Baseline - 22.96 49.88 8.61 38.73 32.53 17.38 8.40 16.79 + Full FT ACL 39.28 65.27 56.32 58.54 52.54 65.97 + Pruning ACL 10.78 39.58 -44.4 42.52 36.92 39.42 6.78 13.55 + FT ACL 32.16 60.39 39.08 53.05 47.23 56.72 + KD 33.44 60.91 39.23 53.41 48.20 54.94 + QLoRA + CV 39.59 65.14 59.21 56.52 50.74 64.34 4.12 8.65 Table 2: Evaluation of the iterative layer pruning experiment. We started by full-parameter fine-tuning (Full FT) of the baseline model Qwen2-Audio-7B-Instruct on the in-domain dataset ACL 60/60. Pruning 8 layers of the decoder of the model achieved approx. 20% reduction in the model size; however, it affected the quality of the model. Hence, we fine-tuned the pruned model again on the in-domain dataset to restore as much as possible of the quality of the fully fine-tuned model. Finally, we fine-tuned the resulting model with low-rank adaptation after quantizing it into the 4-bit precision (QLoRA) on a mix of the in-domain data, knowledge distillation data (KD) and out-of-domain data, namely the CoVoST2 (CV) dataset. The whole process of pruning followed by QLoRA fine-tuning with 4-bit quantization has resulted in approx. 50% reduction in the model size, while retaining 97% and 100% of the translation quality for Chinese and German, respectively, compared to the teacher model. 3 Methodologies We experiment with diverse methods to compress Qwen2-Audio-7B-Instruct while maintaining the translation quality. This section covers our two main experimental setups, while Section 4 discusses several ablation studies and elaborates on intermediate experiments. Our first experimental setup (cf. Section 3.2 ) employs QLoRA fine-tuning with 4-bit quantization, while the other experimental setup (cf. Section 3.3 ) conducts layer importance evaluation and applies iterative layer pruning of the model, followed by QLoRA fine-tuning. Both setups use knowledge distillation to recover the translation quality of the in-domain teacher

Read the complete text