Soundwave : Less is More for Speech-Text Alignment in LLMs
Yuhao Zhang, Zhiheng Liu, Fan Bu, Ruiyu Zhang, Benyou Wang, Haizhou Li · 2025
Read Soundwave : Less is More for Speech-Text Alignment in LLMs by Yuhao Zhang, Zhiheng Liu, Fan Bu, Ruiyu Zhang, Benyou Wang, Haizhou Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12900.txt.
Introduction
Yuhao Zhang and co-authors present Soundwave, a way to train speech language models with far less labeled data, in a 2025 paper. Large speech models often need huge sets of paired speech and text. That cost blocks many teams. Two gaps make the task hard: speech and text live in unlike spaces, and their lengths rarely match.
Soundwave uses a lean training plan and a new model shape to close both gaps. In tests it beat a leading model on speech translation and on a speech task bench, while using only one fiftieth of the training data. More checks show the model keeps its general skill in chat. Code and details are shared on a public project page for others to try.
Catalog record
| CREATOR | Yuhao Zhang, Zhiheng Liu, Fan Bu, Ruiyu Zhang, Benyou Wang, Haizhou Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12900
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Soundwave : Less is More for Speech-Text Alignment in LLMs Yuhao Zhang, Zhiheng Liu, Fan Bu, Ruiyu Zhang, Benyou Wang, Haizhou Li Abstract Existing end-to-end speech large language models (LLMs) usually rely on large-scale annotated data for training, while data-efficient training has not been discussed in depth. We focus on two fundamental problems between speech and text: the representation space gap and sequence length inconsistency. We propose Soundwave, which utilizes an efficient training strategy and a novel architecture to address these issues. Results show that Soundwave outperforms the advanced Qwen2-Audio in speech translation and AIR-Bench speech tasks, using only one-fiftieth of the training data. Further analysis shows that Soundwave still retains its intelligence during conversation. The project is available at https://github.com/FreedomIntelligence/Soundwave . 1 Introduction Large language models (LLMs) have profoundly transformed the paradigm of natural language processing (NLP) due to their remarkable abilities in understanding and reasoning ( Achiam et al., 2023 ; Touvron et al., 2023 ) . Recently, multi-modal LLMs have also shown rapid development, with the success of GPT-4o highlighting the potential of speech-focused LLMs ( Hurst et al., 2024 ) . A fundamental requirement for achieving seamless communication with LLMs is their ability to accurately interpret speech—essentially enabling LLMs to “hear”. Figure 1: AIR-Bench speech foundation tasks. However, most speech-based LLMs rely on massive labeled datasets and substantial computational resources to enable speech perception ( Défossez et al., 2024 ; Chen et al., 2025 ) . For example, the Qwen2-Audio ( Chu et al., 2024 ) model requires approximately 500,000 hours of data to achieve cross-modal functionality, while 1,000 hours needed by advanced automatic speech recognition models to achieve comparable results ( Gulati et al., 2020 ) . This discrepancy underscores the need for more efficient methods to develop speech-capable LLMs. We first identify two fundamental challenges to achieve alignment between speech and text ( Zhang et al., 2023 ) : (1) the representation space gap and (2) sequence length inconsistency . The former challenge arises from the two modalities being developed independently, while the second challenge stems from the difference in modeling units—speech is typically represented at the frame level, whereas LLMs operate at the sub-word level. We then decouple the process of building speech LLMs to achieve more efficient training. Specifically, we propose a two-stage training framework designed to efficiently overcome these challenges. The first stage focuses on resolving the representation space gap, while the second stage aims to reduce the sequence length of speech. Furthermore, to efficiently bridge the gap between speech and text, the quality of alignment data plays a crucial role. To address this, we collect high-quality speech recognition data and manually annotate audio labels to support the first stage. For the second stage, we analyze the proportion of text data to ensure a smooth learning process. During the supervised fine-tuning (SFT) stage, we employ temperature sampling to balance the variety of tasks effectively. We conduct experiments on various speech-to-text tasks and several sound-related tasks. We also compare our model with the strong system Qwen2-Audio on both closed- and open-ended tasks. Our average results achieve state-of-the-art performance on the AIR-Bench ( Yang et al., 2024 ) speech foundation task as shown in Fig. 1 . It also shows comparable results on audio-related tasks. Furthermore, our method exhibits significant performance in zero-shot speech translation, demonstrating that Soundwave unlocks the full potential of LLMs. Soundwave delivers better performance with less training data, lower training costs, and fewer speech sequences. Our main contributions are as follows. 1) We propose an efficient training framework that utilizes only ten thousand hours of training data to achieve state-of-the-art speech understanding performance. 2) We introduce dynamic multi-task learning in the post-training stage to enhance speech modeling and leverage the benefits of text data. 3) We annotate a sound classification dataset to support the alignment between sound and text, and provide high-quality instruction data based on the thinking process for speech LLMs. 2 Methodology Table 1: The parameters of different modules. The orange represents the number of training parameters. Modules #Param. Training Details stage Audio encoder \sim 635M - Whisper Large V3 Alignment \sim 144M I&II One projection and adapter Transformer layer Shrinking \sim 67M II One cross-attention adapter and layer-norm LLMs \sim 8B - Llama3.1 LLM adapter \sim 55M II&III LoRA Total \sim 9B 2.1 Overall Design The training process consists of three stages, as shown in Fig. 2 . Stage I aims to align the representation between speech and text, addressing the representation space gap problem. Stage II primarily shrinks the speech sequence and mitigates the sequence length inconsistency . The supervised fine-tuning (SFT) stage ( Wei et al., 2021 ) enables the speech LLMs to generalize across diverse tasks. The input to the model consists of speech FBank features, which are then processed by the pretrained audio encoder. To efficiently align the representation with that of LLMs, we use the audio encoder that produces semantic features (e.g., Whisper ( Radford et al., 2023 ) or Seamless ( Communication, 2025 ) ), rather than vector quantization features ( Défossez et al., 2022 ) or self-supervised features ( Hsu et al., 2021 ) . We implement an alignment adapter and a shrinking adapter to bridge the gap between speech and text. Additionally, LoRA ( Hu et al., 2022 ) is used to enable efficient fine-tuning. An overview of the modules is provided in Tab. 1 . 2.2 Stage I: Alignment We use the auxiliary CTC loss to improve training efficiency, as it can achieve alignment without the involvement of LLMs. Additionally, we use high-quality data to speed up the convergence rate. 2.2.1 Auxiliary CTC loss The audio encoder and LLMs have a gap in their representation spaces due to separate pre-training. One direct approach is to use ASR tasks for alignment. We design an adapter and utilize CTC loss ( Graves et al., 2006 ) to achieve efficient cross-modal training. Specifically, the adapter consists of a linear layer followed by a Transformer layer ( Vaswani, 2017 ) . The linear layer transforms the audio sequence A\in\mathbb{R}^{l\times h_{a}} where l is the length of the speech sequence and h_{a} is the hidden size of the audio encoder. We concatenate adjacent features and adjust the dimensionality to match that of the LLMs, resulting in A^{\prime}\in\mathbb{R}^{l/2\times h_{llm}} where h_{llm} is the hidden size of the LLMs. A Transformer layer then converts the features into the representation space of LLMs. Finally, we use CTC loss to train the adapter, aligning the shared space of the LLMs. Audio encoder ❄ Speech Alignment adapter … Nice to <blank> Shared LLM embeddings CTC loss (a) Alignment stage Audio encoder ❄ Speech (optional) Alignment adapter CTC loss Shrinking adapter LLMs LoRA Text Embedding ❄ Text instruction ❄ Next token prediction (b) Shrinking stage Audio encoder ❄ Speech Two adapters LLMs LoRA Text Embedding ❄ Text instruction ❄ Next token prediction (c) SFT stage Figure 2: Training progress of Soundwave. The gray modules are frozen while the orange modules are updated. 2.2.2 High-quality Alignment Data We believe that improving data quality is crucial to training efficiency for alignment . We apply data strategies for two types of data (ASR and sound data), as outlined below. The adapter