Optimal Transport Regularization for Speech Text Alignment in Spoken Language Models
Wenze Xu, Chun Wang, Jiazhen Yu, Sheng Chen, Liang Gao, Weihong Deng · 2025
Read Optimal Transport Regularization for Speech Text Alignment in Spoken Language Models by Wenze Xu, Chun Wang, Jiazhen Yu, Sheng Chen, Liang Gao, Weihong Deng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08131.txt.
Introduction
Wenze Xu, Chun Wang, and four co-authors tackle in 2025 a weakness of spoken language models, systems that extend text models to hear speech. These models often do well on the data they were trained on yet stumble on new datasets, even for familiar languages and tasks.
The authors trace the problem to a gap between how speech and text are represented inside the model. Loose speech features let the model lean on accidental quirks instead of word-like structure. Their fix adds an optimal transport penalty that pulls speech and text features into line. Better alignment should help the models treat speech the way they treat text.
Catalog record
| CREATOR | Wenze Xu, Chun Wang, Jiazhen Yu, Sheng Chen, Liang Gao, Weihong Deng |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-08131
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimal Transport Regularization for Speech Text Alignment in Spoken Language Models Thanks: Work done when Wenze Xu interned at Mashang Consumer Finance Co., Ltd. Wenze Xu and Chun Wang contributed equally to this work. Corresponding author: Chun Wang, [email protected] Wenze Xu, Chun Wang, Jiazhen Yu, Sheng Chen, Liang Gao, Weihong Deng Abstract Spoken Language Models (SLMs), which extend Large Language Models (LLMs) to perceive speech inputs, have gained increasing attention for their potential to advance speech understanding tasks. However, despite recent progress, studies show that SLMs often struggle to generalize across datasets, even for trained languages and tasks, raising concerns about whether they process speech in a text-like manner as intended. A key challenge underlying this limitation is the modality gap between speech and text representations. The high variability in speech embeddings may allow SLMs to achieve strong in-domain performance by exploiting unintended speech variations, ultimately hindering generalization. To mitigate this modality gap, we introduce Optimal Transport Regularization (OTReg), a method that formulates speech-text alignment as an optimal transport problem and derives a regularization loss to improve SLM training. In each training iteration, OTReg first establishes a structured correspondence between speech and transcript embeddings by determining the optimal transport plan, then incorporates the regularization loss based on this transport plan to optimize SLMs in generating speech embeddings that align more effectively with transcript embeddings. OTReg is lightweight, requiring no additional labels or learnable parameters, and integrates seamlessly into existing SLM training procedures. Extensive multilingual ASR experiments demonstrate that OTReg enhances speech-text alignment, mitigates the modality gap, and consequently improves SLM generalization across diverse datasets. 1 Introduction Large Language Models (LLMs) [ 6 , 18 ] have significantly advanced text-based language understanding and generation. Their success has driven progress in speech understanding research, leading to efforts to extend LLMs into Spoken Language Models (SLMs) [ 2 , 20 ] , which can directly process speech input—an essential and natural aspect of language representation. One practical approach to building SLMs involves integrating a pretrained speech encoder with an LLM through an adapter module [ 1 ] . The adapter functions as a bridge, transforming speech embeddings into a representation compatible with the LLM’s input space, thereby enabling direct speech input processing. This method leverages the strengths of existing components, allowing the SLM to process speech inputs while preserving the LLM’s advanced language understanding capabilities. Previous studies have shown that SLMs employing this approach achieve competitive performance in major speech understanding tasks, such as automatic speech recognition (ASR) [ 14 ] and speech-to-text translation [ 24 ] , validating the effectiveness of adapter-based integration in SLM development. Despite advancements in in-domain datasets, recent studies [ 7 , 12 ] indicate that SLMs experience significant performance degradation across datasets, even when the spoken languages and tasks were included in training. These findings raise concerns that SLMs may not truly comprehend speech in a text-like manner as intended. Instead, their strong performance on in-domain data may partially result from exploiting unintended speech variations, suggesting overfitting [ 12 ] . Addressing these limitations is crucial for enhancing generalization and unlocking the full potential of SLMs in diverse speech applications. Figure 1: Challenges in speech-text alignment. Recent studies [ 7 , 13 , 22 ] suggest that the modality gap between speech and text is a key factor in this challenge. Because of high frame rates, speech embeddings are often substantially longer than their corresponding transcript embeddings. Furthermore, unlike transcript embeddings, which primarily encode linguistic content, speech embeddings also incorporate paralinguistic features such as pauses and variations in speech rate, adding complexity to their representation. These differences make speech more dynamic than text, increasing the likelihood that SLMs will capture irrelevant variations in speech rather than focusing solely on linguistic content. Addressing these issues requires precise speech-text alignment to reduce the modality gap, thereby enhancing SLM generalization. Achieving precise speech-text alignment during SLM training presents two key challenges. As illustrated in Fig. 1 , the first challenge is that while pairing speech with its transcript is straightforward, defining and establishing meaningful pointwise correspondences between heterogeneous speech and transcript embeddings remains difficult. The second challenge is designing a differentiable and robust loss function that optimizes the speech model to generate speech embeddings aligned with transcript embeddings based on the established pointwise correspondences. This is particularly challenging because dynamically established correspondences are often noisy and tend to change during training. In this paper, to tackle these challenges, we introduce Optimal Transport Regularization (OTReg) as a novel approach to bridging the modality gap in SLMs. We formulate speech-text alignment as an optimal transport (OT) problem [ 21 ] , treating speech embeddings as the source and transcript embeddings as the target. In this framework, pointwise correspondences between the source and target are represented by a transport plan. Given a predefined cost matrix, OT determines the optimal transport plan by solving an optimization problem that minimizes the total transport cost. To encourage correspondences between similar embeddings, we define the transport cost based on cross-modal embedding similarity, ensuring that lower transport costs occur when embeddings exhibit higher similarity. Furthermore, to train SLMs to generate speech embeddings that better align with transcript embeddings, we derive a regularization loss from the optimal transport plan and incorporate it into the SLM training process. Each training iteration consists of two steps: first, estimating the optimal transport plan based on speech embeddings generated using the current SLM parameters; second, computing a regularization loss derived from the estimated transport plan and integrating it with the primary loss, jointly optimizing the SLM through backpropagation. The proposed OTReg directly employs transcript embeddings as alignment targets, requires no additional annotated labels, and introduces no extra learnable parameters, ensuring seamless integration into the SLM training framework. Experimental results on multilingual ASR tasks demonstrate that OTReg effectively enhances speech-text alignment, reduces the modality gap, and consequently improves the generalization of SLMs across diverse datasets. The remainder of this paper is structured as follows: Sec. 2 covers preliminaries, then Sec. 3 reviews related work. The proposed method is detailed in Sec. 4 . Then, Sec. 5 and Sec. 6 present the experimental setup and results, respectively. Finally, Sec. 7 concludes the paper. 2 Preliminaries In this section, we provide a concise recap of CTC and Optimal Transport to facilitate understanding of related work and this study. 2.1 Connectionist Temporal Classification Connectionist Temporal Classification (CTC) loss [ 10 ] is widely used in speech processing tasks to align speech embeddings with textual labels when annotated pointwise correspondence is unavailable. Instead of relying on predetermined alignments, CTC formulates the alignment problem by predicting a probability distribution over all possible label sequences derived from speech embeddings. The training objective is to maximize the cumulative probability of all sequences that, when appropriately collapsed, match the ground truth label sequences. To handle non-informative segments such as silence or pauses in speech, CTC introduces a special “blank” label (denoted as “-”), allowing the model to accommodate timing and duration variability. CTC-based compression [ 9 ] extends CTC to reduce redundancy in speech embeddings while preserving essential linguistic information. It leverages CTC predictions to merge consecutive embeddings corresponding to the same label and discard non-informative embeddings associated with the blank label.