The Eloquence team submission for task 1 of MLC-SLM challenge
Lorenzo Concina, Jordi Luque, Alessio Brutti, Marco Matassoni, Yuchen Zhang · 2025
Read The Eloquence team submission for task 1 of MLC-SLM challenge by Lorenzo Concina, Jordi Luque, Alessio Brutti, Marco Matassoni, Yuchen Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19308.txt.
Introduction
Lorenzo Concina and co-authors describe their entry to a multilingual speech challenge, in a 2025 paper. The task was to recognize speech in many tongues inside real conversations. Such skill matters for spoken dialogue systems that serve global users. The team tried three approaches in turn.
First, they probed the official baseline, training two kinds of connector between speech and language models. Next, they built a custom multilingual connector in a known framework. Last, they tested whether contrast methods and longer conversation context made recognition more robust. Challenge entries of this kind let the field compare recipes on shared data. The paper reports what each recipe gained and where it fell short.
Catalog record
| CREATOR | Lorenzo Concina, Jordi Luque, Alessio Brutti, Marco Matassoni, Yuchen Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 16 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-19308
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Eloquence team submission for task 1 of MLC-SLM challenge Lorenzo Concina, Jordi Luque, Alessio Brutti, Marco Matassoni, Yuchen Zhang Abstract In this paper, we present our studies and experiments carried out for the task 1 of the Challenge and Workshop on Multilingual Conversational Speech Language Model (MLC-SLM), which focuses on advancing multilingual conversational speech recognition through the development of speech language models architectures. Given the increasing relevance of real-world conversational data for building robust Spoken Dialogue Systems, we explore three approaches to multilingual ASR. First, we conduct an evaluation of the official baseline to better understand its strengths and limitations, by training two projectors (linear and qformer) with different foundation models. Second we leverage the SLAM-ASR framework to train a custom multilingual linear projector. Finally we investigate the role of contrastive learning and the extended conversational context in enhancing the robustness of recognition. † † address: 1 Fondazione Bruno Kessler 2 Telefónica Innovación Digital, Scientific Group 3 University of Essex † † email: {lconcina, brutti, matasso}@fbk.eu [email protected] [email protected] Index Terms : multilingual speech recognition, SpeechLLM 1 Introduction Large Language Models have revolutionized natural language processing by enabling powerful and flexible systems for a wide range of text-based tasks. Extending these capabilities to spoken language is a next step toward building a multimodal human-computer interaction system. This has led to the emergence of Speech Language Models (SLMs) [ 1 ] , which aim to unify speech and language understanding within a single model architecture by extending LLMs to the audio modality. This approach aims to solve several tasks that were previously text-based in the audio domain. Among these are Automatic Speech Recognition, Spoken Language Understanding, and Emotion Recognition. Traditionally, spoken language understanding systems have relied on cascaded pipelines, where ASR is followed by text-based language modeling and, in some cases, text-to-speech (TTS). However, this approach often suffers different issues: error propagation, lack of contextual integration between components, and the loss of paralinguistic cues and contextual dependencies due to the fact that the LLM only processes the transcription of the speech. In this work, we describe our participation in the MLC-SLM Challenge to explore the development of multilingual conversational SLMs. Our focus is on building models that can operate directly on speech data in multiple languages, leveraging LLM-based architectures and the challenge data to improve recognition quality in complex, real-world dialogue settings. Additionally, through our participation, we aim to help disseminate the goals and ongoing research within the EU-funded project Eloquence 1 1 1 https://eloquenceai.eu/ , which focuses on advancing multimodal, multilingual, and multi-task speech technologies [ 2 , 3 ] . 2 Description of the system In this study we faced the ASR task of the challenge with the Speech and Language Model (SLM) [ 4 ] approach, leveraging the power of an LLM to compute the transcription of the input audio, which is first passed to an audio encoder, then downsampled and finally projected by a linear projector to match the dimension of the input embeddings of the LLM. This approach is showed in figure 1 . The LLM input is the concatenation of the processed input speech, the prompt (which undertake only the tokenization step) and, at training time, the reference transcription. In this system, the critical element is the projector, which must be trained while keeping the encoder and LLM frozen. Furthermore, it’s possible to finetune the LLM together with the projector, or eventually use LoRA in case of an insufficient amount of training data. The projector and the LLM can be fine-tuned jointly, as demonstrated in our SLAM setup, or they can be fine-tuned in two separate stages, as implemented in the challenge pipeline. Our experiments indicate that the joint fine-tuning approach generally yields better performance. Figure 1: SLM pipeline In the challenge, we explored 3 different directions that are described in the following sections: • we experimented with the challenge codebase provided as baseline 2 2 2 MLC-SLM baseline ; • we investigated the use of the SLAM-ASR [ 5 ] framework • we explored the use of providing previous context as well as contrastive learning solutions. 2.1 Baseline Evaluation For comparison of the results, we experimented with the baseline code provided by the organizers. The MLC-SLM baseline is implemented using the Wenet [ 6 ] Recognition Toolkit. The ASR model is composed of Whisper-large-v3 as speech encoder and Qwen2.5-7B or Llama3.1-8B as the backbone LLM. The training consists of two stages. In the first step a projector, composed of a convolutional layer with down-sampling factor 4 and a linear layer, is trained between the speech encoder and the LLM. In the following step, it simultaneously trains both the projector and LLM, the latter using Low-Rank Adaptation (LoRA), but starting from the projector trained in the first step. The table 2 reports the result of the baseline submitted but using Whisper-large-v3 Turbo instead of the proposed baseline encoder. From the development experiments, the outcomes did not reveal a substantial difference when employed the two-stage training strategy. Consequently, we opted to submit additional trials wherein just the second training step was applied, that is, only training simultaneously the projector and the LLM. Similar findings were observed in the context of switching the speech encoder, wherein using Whisper-large-v3 Turbo did not yield significant impact. As reported in Table 2 , we also compared several LLMs within the baseline Qwen2.5-7B: EuroLLM 1.7B and Salamandra with 2B and 7B parameters versions. For LoRA we follow the default settings provided by the baseline, with rank r set to 16 and scaling factor \alpha=8 . All speech encoders are feed with 128 log_mel features and the training dataset is augmented using spectral masking augmentation and speed perturbation based on the SpecAugment [ 7 ] , with default values as in the baseline provided recipe. The table also reports the results of a Querying Transformer (Q-Former) based projector. It is based on the Blip2QFormer implementation from the SLAM-ASR [ 5 ] code. The Q-Former takes as input the entire output sequence of length 1280 from the Whisper encoder without down-sampling and applies two Q-Former hidden layers with a query length of 64. 2.2 SLAM-ASR The SLAM-ASR [ 5 ] framework is a multilingual Speech Language Model (SLM) that extends a pretrained Large Language Model (LLM) to process spoken input directly, following the architecture shown in figure 1 . Although this framework presents yet unclear performances against issues such as being robust enough across different scenarios and speech conditions, such as domain shifts and speech perturbations [ 8 ] , it is a powerful tool to experiment with different combinations of pre-trained foundational models and hyperparameters to achieve strong ASR capabilities. As the first line of experiments, we decided to leverage this tool to train a linear projector with different combinations of foundational models and hyperparameterers. Table 1 shows a list of the main experiments and the average WER computed in the challenge evaluation set. These experiments led to our best system, which is composed of Whisper Large V3 Turbo as audio encoder, a linear projector, and EuroLLM 1.7 [ 9 ] (frozen during training, but to which we apply LoRA). This experiment is the one appearing on the challenge public leaderboard, outperforming