Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving

Jingran, Xiang, Hui, Yue, Yang, Xixin, Zhiyong · 2025

Read Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving by Jingran, Xiang, Hui, Yue, Yang, Xixin, Zhiyong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18644.txt.

Introduction

Jingran Xie and co-authors present a training method to help speech language models generalize to new tasks. Their 2025 paper notes that speech models are usually fine-tuned to match a text model, but labeled speech data are scarce for many tasks. That shortage makes alignment slow and leaves models brittle on fresh prompts.

Their method, called MTBI, pairs multi-task behavior imitation with interleaved speech and text. It needs only speech recordings with transcripts. The model is trained so its decoder answers speech and text inputs in the same way. The team also built a benchmark for prompt and task generalization. Their model beat leading rivals on both kinds, while using less labeled speech.

Catalog record

CREATORJingran, Xiang, Hui, Yue, Yang, Xixin, Zhiyong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18644

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Generalization of Speech Large Language Models with Multi-Task Behavior Imitation and Speech-Text Interleaving Jingran, Xiang, Hui, Yue, Yang, Xixin, Zhiyong Abstract Large language models (LLMs) have shown remarkable generalization across tasks, leading to increased interest in integrating speech with LLMs. These speech LLMs (SLLMs) typically use supervised fine-tuning to align speech with text-based LLMs. However, the lack of annotated speech data across a wide range of tasks hinders alignment efficiency, resulting in poor generalization. To address these issues, we propose a novel multi-task ’behavior imitation’ method with speech-text interleaving, called MTBI, which relies solely on paired speech and transcripts. By ensuring the LLM decoder generates equivalent responses to paired speech and text, we achieve a more generalized SLLM. Interleaving is used to further enhance alignment efficiency. We introduce a simple benchmark to evaluate prompt and task generalization across different models. Experimental results demonstrate that our MTBI outperforms SOTA SLLMs on both prompt and task generalization, while requiring less supervised speech data. Xie Li Wang Yu Xiang Wu Wu Tsinghua UniversityChina Pengcheng LaboratoryChina The Chinese University of Hong KongChina keywords speech large language model, speech-text alignment, supervised fine-tuning † † email: {xjr21, lix23}@mails.tsinghua.edu.cn, {wangh06, yuy, xiangy}@pcl.ac.cn, {wuxx, zywu}@se.cuhk.edu.hk 1 1 footnotetext: Corresponding author. 1 Introduction In recent years, large language models (LLMs) [ 1 , 2 ] have demonstrated remarkable capabilities, particularly in zero-shot scenarios, showcasing their ability to generalize across a wide range of tasks. To leverage this ability for speech, substantial progress [ 3 , 4 , 5 , 6 , 7 ] has been made in the development of speech large language models (SLLMs), which process spoken inputs and generate text responses following textual task prompts. The common approach [ 4 , 5 , 6 ] in developing these models involves aligning speech with the text embedding space of LLMs through supervised fine-tuning (SFT) with labeled speech data. This alignment enables the LLM to understand speech content and leverage its generalization ability to handle various tasks based on speech input. Due to the limited scale of speech data, some tricks like multi-task learning [ 8 , 9 ] and multi prompt training [ 6 ] are often used to enhance SLLM’s generalization. However, despite promising results, some researchers [ 10 ] have noted that the SFT-based approach degrades the model’s ability to follow text prompts effectively. Our experiments also indicate that SLLMs tend to exhibit poorer generalization capabilities compared to purely text-based models on zero-shot tasks (Unseen task during SFT). We believe this limitation arises from the scarcity of annotated speech data. Even the most advanced SLLM, such as Qwen2-audio [ 6 ] , is SFT on a limited number tasks, which are far fewer than the wide range encountered in real-world scenarios. The inherent difficulty and cost of annotating speech data across diverse tasks make such extensive coverage unlikely. To address these challenges, we draw inspiration from successful SLLMs [ 10 , 6 , 11 , 12 ] , and propose a novel method that leverages multi-task ”behavior imitation” training with interleaving, referred to as MTBI. Our motivation is to achieve a high level of alignment between speech and text modalities on easily accessible ASR data, without reliance on large amount of hard-to-obtained supervised data. This well alignment allows the SLLM to fully leverage the generalization ability of the LLM. We train the SLLM to generate responses equivalent to the text-based LLM when given the same speech and transcription, just like training SLLMs to imitate the behavior of text LLM. Additionally, we also incorporate multi-task learning method, which performs behavior imitation across diverse task distributions, to further enhance the generalization. Inspired by recent text-to-speech (TTS) work [ 12 , 13 ] , we introduce an interleaving technique that mix speech and text as inputs, effectively improving the alignment between two modalities. Given the lack of attention to the generalization abilities of SLLMs, we follow the natural language processing (NLP) generalization evaluation methods [ 14 , 15 , 16 ] , to present a simple benchmark to evaluate the generalization capabilities of SLLMs, assessing their adaptability to various prompts and zero-shot tasks. In summary, our contributions are as follows: • We propose a novel multi-task behavior imitation training method that leverages easily accessible transcriptions, reducing the reliance on extensive annotated speech data. This method significantly enhances the zero-shot generalization capability of SLLMs, enabling them to perform effectively across a wide range of tasks. • We introduce an interleaving technique that combines speech and text as inputs. This enhances the alignment efficiency between speech and text modalities, greatly improving the generalization of the SLLM. • We design a benchmark to systematically evaluate SLLMs’ generalization to different prompts and zero-shot tasks, providing a clear framework for comparing model performance and advancing the field. Through experiments, we demonstrate that our MTBI achieves or exceeds the performance of SOTA SLLMs in terms of prompt and task generalization. Figure 1: Overview of the SLLM architecture. The training process is conducted in two stages using the same text LLM. In stage 1, we use the LLM to generate responses based on the task prompts and transcripts of speech data. Then, we train the SLLM model with behavior imitation that use the same task prompt and corresponding speech (or interleaved speech) to predict the generated response of the first stage. In stage 2, we only train the connector to align the speech features into textual space. 2 Methodology 2.1 Model Architecture We train MTBI on a simple and popular vanilla speech LLM framework [ 5 , 6 , 17 ] as shown in Figure 1 , right part. This architecture integrates an frozen speech encoder, a trainable modality connector and a frozen text LLM to generate response. Speech Encoder The speech encoder is responsible for converting raw audio signals into a meaningful representation that contains well-organized linguistic and acoustic features. We use the WavLM Large [ 18 ] as the speech encoder, which consists of 24 Transformer layers. We obtain the speech representation from the last layer. To reduce the training costs, we froze the speech encoder throughout the entire training. Connector We introduce a small trainable connector [ 19 , 20 ] to link the speech and text modalities, which can efficiently map speech representation to the textual space of LLM with minimal computational cost. Since speech embeddings are typically much longer than text embeddings, we use a CNN based subsampler connector to reduce the length discrepancy between two modalities. This connector functions as an intermediary between the speech encoder and the LLM decoder, efficiently fusing speech embeddings with textual input and facilitating smooth interactions between the two modalities. LLM Decoder To generate text responses, we employ LLaMA2-7B-chat [ 2 ] as the LLM decoder, which is chosen for its strong conversational abilities. The decoder receives text task prompts and processed speech embeddings from the encoder and connector, then generates a coherent output. To prevent catastrophic forgetting of internal knowledge, LLM decoder is frozen throughout the training process. 2.2 Behavior Imitation 2.2.1 Behavior Imitation Training As discussed in the introduction, our objective is let the LLM decoder produce consistent responses when presented on speech and its corresponding transcription. Achieving this consistency would indicate

Read the complete text