NAVER LABS Europe Submission to the Instruction-following Track
2025
Read NAVER LABS Europe Submission to the Instruction-following Track (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01808.txt.
Introduction
Beomseok Lee and three colleagues at NAVER LABS Europe describe their entry to a speech instruction-following contest. Their report appeared in twenty twenty-five. The IWSLT twenty twenty-five short track asked one system to transcribe, translate, and answer questions from English speech.
Working under the constrained rules, the team fused two pretrained parts. A projector maps speech features from a large multilingual encoder into a language model's space. Small adapter layers, trained on text, tune an eight-billion-parameter model for the tasks. Joint instruction tuning on mixed speech and text data ran for one thousand steps. Target languages were Chinese, Italian, and German. The report documents the full recipe behind the submitted system.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01808
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NAVER LABS Europe Submission to the Instruction-following Track Abstract In this paper we describe NAVER LABS Europe submission to the instruction-following speech processing short track at IWSLT 2025. We participate in the constrained settings, developing systems that can simultaneously perform ASR, ST, and SQA tasks from English speech input into the following target languages: Chinese, Italian, and German. Our solution leverages two pretrained modules: (1) a speech-to-LLM embedding projector trained using representations from the SeamlessM4T-v2-large speech encoder; and (2) LoRA adapters trained on text data on top of Llama-3.1-8B-Instruct . These modules are jointly loaded and further instruction-tuned for 1K steps on multilingual and multimodal data to form our final system submitted for evaluation. \affiliations 1 NAVER LABS Europe, France 2 University of Trento, Italy 3 Fondazione Bruno Kessler, Italy \contributions ⋆ equal contribution \teaserfig \teasercaption Our training pipeline. A speech projector (A) and text LoRA adapters (B) are trained in parallel using speech-to-text and text-to-text data, respectively. These modules are then integrated during a brief multimodal adaptation step (C). † † authors: Beomseok Lee 123⋆ , Marcely Zanon Boito 1⋆ , Laurent Besacier 1 , Ioan Calapodescu 1 1 Introduction Large language models (LLMs) have demonstrated remarkable success across various text-based natural language processing tasks [ 2 , 32 , 13 , 34 , 3 , 19 ] , motivating research into extending them to other modalities. This has led to the development of multimodal LLMs capable of processing speech, audio, images and video [ 29 , 7 , 26 , 17 , 28 , 6 , 10 , 14 , 11 , 20 , 4 ] . This year IWSLT Instruction-following Speech Processing Track focuses on the leveraging of LLMs and speech foundation models (SFM) to build solutions capable to perform multilingual tasks from English speech input and textual multilingual instructions [ 1 ] . NAVER LABS Europe (NLE) participates in the constrained setting of the short track , where the tasks proposed are automatic speech recognition (ASR), speech translation (ST) and multilingual spoken question answering (SQA). The target languages for ST and multilingual SQA are Chinese, Italian and German. The participants are allowed to use the speech backbone SeamlessM4T-v2-large [ 5 ] and the text LLM Llama-3.1-8B-Instruct [ 8 ] for both training and data generation. Our submitted systems leverage all the available data from the constrained settings, together with data automatically obtained using both backbones. We train two types of systems in parallel: (1) speech-to-text ASR/ST/SQA projectors that project the averaged speech representation from the SeamlessM4T-v2-large encoder to the embedding space of a frozen Llama-3.1-8B-Instruct ; (2) text-only LoRA adapters [ 9 ] , plugged on top of the same frozen LLM. Once both systems are separately trained, we show that we can merge them, increasing overall speech performance, by fine-tuning for only 1K steps on multimodal multilingual data. This system paper is organized as follows. Section 2 describes the preprocessing applied to the data used in this challenge. Sections 3 and 4 describe our training pipeline and experimental settings, respectively. Section 5 presents our experiments and discussion. Section 6 presents the submitted system. Section 7 concludes the paper. 2 Data For training our models, we leverage the data from the constrained setting: CoVoST2 [ 33 ] , EuroParlST [ 12 ] and SpokenSQuAD [ 15 ] . With the agreement of the organizers, we also take advantage of the SeamlessM4T-v2-large to produce extra synthetic speech data (Seamless TTS) and multilingual text data (Seamless MT). Llama-3.1-8B-Instruct is used to rephrase SQA answers. ACL 60-60 [ 27 ] is used for validation and evaluation only. We now present our data preprocessing (Section 2.1 ) and prompt format (Section 2.2 ). 2.1 Data Preprocessing We produce both speech-to-text and text-to-text instructions to train our systems. For the aforementioned datasets, we produce the following splits, where * denotes synthetic splits obtained via SeamlessM4T-v2-large MT; {\dagger} indicates splits generated with SeamlessM4T-v2-large TTS; and {\ddagger} marks those derived through Llama-3.1-8B-Instruct -based rephrasing. • CoVoST2: – ASR and ST/MT (en-de; en-zh) • EuroParlST: – ASR and ST/MT (en-de; en-it) • SpokenSQuAD: – ASR † and MT (en † -de*; en † -it*; en † -zh*) – SQA/QA (en † -en; en † -de*; en † -it*; en † -zh*) – fluent SQA/QA (en † -en ‡ ; en † -de ∗,‡ ; en † -it ∗,‡ ; en † -zh ∗,‡ ) • ACL 60-60: – ASR and ST/MT (en-de; en-it*; en-zh) Below we detail dataset-specific preprocessing. Statistics are presented in Table 1 . CoVoST2 and EuroParlST CoVoST2 covers English to German and Simplified Chinese language directions. EuroParlST covers English to German and Italian. ASR splits for these datasets were built by merging the existing language splits and deduplicating the audio files. For both, language-specific ST and MT splits are created by aligning translations to English speech and reference transcriptions, respectively. SpokenSQuAD The SpokenSQuAD dataset is organized into two jsons, train and test. Each split is organized in themes, each with several paragraphs. For each paragraph, TTS audio files are available, aligned at the sentence level. 1 1 1 The format is themeID \_ paragraphID \_ sentenceID. For each sentence, questions (each with several answers) are available. We performed the following modifications to this dataset: • Duplicated answers: we removed duplicated answers to the same question using exact string matching, as well as any questions that required more than one audio file to answer, since we are participating in the SHORT track. • Validation set: we created a validation set by selecting the first 20 themes of the training set and removing them from training (3,102 entries). • New TTS audio: we generate new TTS audio files for the training set using SeamlessM4T-v2-large . Resynthesizing the source audio for this dataset was necessary due to existing dataset misalignment, which we detail in Appendix Section A.1 . • Multilingual SQA/QA: we created multilingual SQA/QA sets by first translating questions and answers to target languages using SeamlessM4T-v2-large . We then use reference-free COMET 2 2 2 Unbabel/wmt22-cometkiwi-da [ 25 ] to filter out all pairs of questions and answers that do not both score at least 0.85. • Invalid splits: we created the invalid SQA sets by deliberately mismatching context and question themes, thereby creating unanswerable examples. The corresponding answers were labeled as “Not answerable” in four languages (English, Italian, German and Chinese), following the guideline answer provided by the task organizers. While we acknowledge that a small, unknown subset of these reassigned questions may still be answerable, we hypothesize that ensuring a thematic mismatch between the reference context and the question is the most effective strategy for minimizing this issue. • Fluent SQA/QA version: we created an alternative SQA/QA training set using Llama-3.1-8B-Instruct to regenerate the dataset original answers as fluent sentences. The motivation behind this was the observation that, since the original answers are made of an exact extract of the reference audio/text, the model had a difficult time answering some out-of-domain questions fluently. 3 3 3 In the official IWSLT 2025 test set we observed examples as the following. Question: “What are the names of the speakers?” Our model’s answer: “yin and my colleague jiang” . While the model answer is