Instituto de Telecomunicações at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning
Giuseppe Attanasio ♣, Sonal Sannigrahi ♣ ♠, Ben Peters ♣, André F.T. Martins ♣ ♠ ♢ · 2025
Read Instituto de Telecomunicações at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning by Giuseppe Attanasio ♣, Sonal Sannigrahi ♣ ♠, Ben Peters ♣, André F.T. Martins ♣ ♠ ♢ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17019.txt.
Introduction
Giuseppe Attanasio and co-authors report their entry to a shared speech task in 2025. Their system turns speech into text for three jobs at once: writing down what was said, translating it, and answering spoken questions.
The design joins a trained speech reader to a small text model in two steps. First the two parts learn to line up, then the joined model learns to follow orders. The team stress small size and clean data rights, using open data plus made up data to fill gaps. The report shows how a compact system can still handle all three jobs.
Catalog record
| CREATOR | Giuseppe Attanasio ♣, Sonal Sannigrahi ♣ ♠, Ben Peters ♣, André F.T. Martins ♣ ♠ ♢ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-17019
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Instituto de Telecomunicações at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning Giuseppe Attanasio ♣, Sonal Sannigrahi ♣ ♠, Ben Peters ♣, André F.T. Martins ♣ ♠ ♢ Abstract This paper presents Instituto de Telecomunicações’s submission to the IWSLT 2025 Shared Task on Instruction Following Speech Processing. We submit results for the Short Track, i.e., speech recognition, translation, and spoken question answering. Our model is a unified speech-to-text model that integrates a pretrained continuous speech encoder and text decoder through a first phase of modality alignment and a second phase of instruction fine-tuning. Crucially, we focus on using small-scale language model backbones (< 2B) and restrict to high-quality, CC-BY data along with synthetic data generation to supplement existing resources. 1 1 1 Code and data at https://github.com/deep-spin/it-iwslt-2025 . 1 Introduction This paper presents our submission to the IWSLT 2025 Instruction Following track for the tasks of Automatic Speech Recognition (ASR), Speech Translation (ST), and Spoken Question Answering (SQA) for English, Chinese, and German. Our work builds upon a long line of previous research equipping LMs with additional multimodal capabilities, aligning an LM’s semantic spaces with that of a pretrained speech encoder ( Tang et al. 2023 ; Huang et al. 2023 ; Hu et al. 2024 ; Chu et al. 2024 ; Grattafiori et al. 2024 , inter alia) . Our contribution is particularly motivated by efficiency , i.e., the goal of achieving strong performance using small-scale (< 2B) models. Recent work has explored audio quantization techniques ( Zhang et al. 2024 ; Défossez et al. 2023 , inter alia) , quantized input mel spectrograms Shen et al. 2024 , or extreme compression of input data over the time dimension through convolutional kernels paired with strong small-scale LM backbones Abouelenin et al. 2025 . We take stock of such advancements and propose a model for even smaller scales. We use established methods for speech integration in LMs ( Gaido et al. 2024 ; Grattafiori et al. 2024 ) using pretrained base models of up to 1.5B learnable parameters, finding empirically that with highly filtered and synthetic data, we can enable similar results at a fraction of the cost of larger LMs. The main contributions of our system are as follows: • Adapting pretrained, small-scale LMs: We experiment with Qwen 2.5 1.5B and 0.5B Yang et al. 2024a as our LM of choice and use w2v-BERT 2.0 Barrault et al. 2023 as our speech encoder. • Two-stage Training Curriculum: We use a modality alignment and instruction fine tuning (IFT) phase for training our models, where the first equips the model with general speech capabilities and the second enables multi-task capabilities. • Training on open-licensed data: To guarantee reproducibility and facilitate future research, we train on established CC-BY data collections and synthetic data filtered for quality. We release training and modeling artefacts under a permissive license. 2 Related Work Efficient LMs. Recent works on efficient, small-scale language models (SLMs) have shown impressive knowledge compression capabilities by maintaining similar performance to larger, more computationally-intensive models. Models such as Phi-4 Mini Abouelenin et al. 2025 and Gemma 2 Gemma Team et al. 2024 have reported strong performance relative to size with a focus on computational efficiency. Lu et al. 2024 have shown how scaling laws operate differently for SLMs and have further demonstrated the efficiency of such models in subsequent reasoning tasks. Multimodal LM Extension. Equipping a text-based model with multimodal capabilities is often done using an auxiliary modality encoder that is then used to jointly learn a semantic mapping between speech and text. Early works in joint text-speech modeling include AudioPaLM Rubenstein et al. 2023 , VioLA Wang et al. 2023 , and VoxtLM Maiti et al. 2024 . Other approaches combine a pretrained continuous speech encoder with an LM by concatenating speech embeddings to the text context ( Tang et al. 2023 ; Huang et al. 2023 ; Hu et al. 2024 ; Chu et al. 2024 ; Grattafiori et al. 2024 , inter alia) . Such works rely on strong multilingual capabilities of the speech encoder and those of large-scale LMs (i.e., 7B or more) to learn how to use speech-related parts of the context ( Grattafiori et al. 2024 ) . Our system echoes this compositional approach to speech and language modeling but leverages recent language models in the scale 0.5-1.5B. 3 System Overview 3.1 Model Architecture Our model follows a standard speech encoder, text decoder architecture ( Tang et al. 2023 ; Grattafiori et al. 2024 ; Chu et al. 2024 , e.g.,) . Speech stack. We extract 80-dimensional Mel-filterbank audio representations with a stride of 2 using w2v-BERT 2.0’s standard processor. Then we compress the audio over the time dimension using three 1D convolutional layers with a kernel width of 3 and a stride of 2. This input is then processed by the pretrained w2v-BERT 2.0 model. The output representations are processed by a modality and length adapter composed of two Conformer-like ( Gulati et al. 2020 ) layers that further compress the audio representations on the time dimension and project them into the embedding space of the language model. Text stack. We prepend the audio representations computed from the audio stack to the text input embeddings extracted from the input embedding matrix of the language model. We use a bidirectional self-attention for the audio positions and a causal (autoregressive) one for the text part of the context. Following prior work Chu et al. 2023 ; Radford et al. 2022 , we constrain text generation using a target language and task token. Figure 1 illustrates the model architecture. Figure 1: Illustration of our model. During training, the speech stack (red) generates speech representations which are prepended to task and language tags and an (optional) length hint (yellow), and text tokens (green). At inference time, we only provide the language and task tokens. 3.2 Training Curriculum We train our model in two stages: a modality alignment stage followed by an instruction fine-tuning (IFT) stage. Modality Alignment. This first stage aligns the speech stack output representations to the language model’s embedding space. We use the pretrained w2v-BERT 2.0 (v2) 2 2 2 https://huggingface.co/facebook/w2v-bert-2.0 as our speech encoder and randomly initialize the pre-encoder convolutional layers and the modality adapter. We choose Qwen 2.5 1.5B, 3 3 3 https://huggingface.co/Qwen/Qwen2.5-1.5B a multilingual LM, as our text decoder. Choosing a small (< 2B) model allows for the exploration of more efficient alternatives and is often overlooked in the literature. In this phase, we train only the pre-encoder convolutional layers and the modality adapter with a learning rate of 3\times 10^{-3} for a single epoch. We train the model only on ASR data. The model is trained using standard cross-entropy loss on the reference transcript tokens. With a 95% chance, we prepend to the language and task tags a length hint , as suggested by Deitke et al. 2024 , to let the model learn a length distribution. This stage leads to a model that can perform ASR but does not yet have other capabilities. Instruction Fine-Tuning. Following the modality alignment phase, we perform IFT using speech-to-text tasks included in the IWSLT campaign (AST, SQA) in English, German, and Chinese. During this