POTSA: A Cross-Lingual Speech Alignment Framework for Low Resource Speech-to-Text Translation
Xuanchen Li, Chenrui Cui, Tianrui Wang, Meng Ge, Zikang Huang, Jin Li, Yizhou Peng [-0.25em] Longbiao Wang, Jianwu Dang, Nyima Tashi · 2025
Read POTSA: A Cross-Lingual Speech Alignment Framework for Low Resource Speech-to-Text Translation by Xuanchen Li, Chenrui Cui, Tianrui Wang, Meng Ge, Zikang Huang, Jin Li, Yizhou Peng [-0.25em] Longbiao Wang, Jianwu Dang, Nyima Tashi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-09232.txt.
Introduction
Xuanchen Li leads a twelve-author team tackling uneven speech translation quality across languages. Speech large language models translate well for major languages, but they underuse semantic common ground shared across source languages. The result is biased performance that favors high-resource languages.
Their framework, POTSA, aligns speech representations using parallel speech pairs and optimal transport. A bias compensation module first aligns representations coarsely, then token-level constraints on a Q-Former enforce fine-grained consistency. A layer scheduling strategy applies those constraints only where they help semantically. On the FLEURS benchmark, POTSA sets new state-of-the-art results. It gains just over one BLEU point on common languages and nearly three on zero-shot languages. Training used only ten hours of parallel speech per language.
Catalog record
| CREATOR | Xuanchen Li, Chenrui Cui, Tianrui Wang, Meng Ge, Zikang Huang, Jin Li, Yizhou Peng [-0.25em] Longbiao Wang, Jianwu Dang, Nyima Tashi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-09232
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
POTSA: A Cross-Lingual Speech Alignment Framework for Low Resource Speech-to-Text Translation Xuanchen Li, Chenrui Cui, Tianrui Wang, Meng Ge, Zikang Huang, Jin Li, Yizhou Peng [-0.25em] Longbiao Wang, Jianwu Dang, Nyima Tashi Abstract Speech Large Language Models (SpeechLLMs) have achieved breakthroughs in multilingual speech-to-text translation (S2TT). However, existing approaches often overlook semantic commonalities across source languages, leading to biased translation performance. In this work, we propose POTSA (Parallel Optimal Transport for Speech Alignment), a new framework based on cross-lingual parallel speech pairs and Optimal Transport (OT), designed to bridge high- and low-resource translation gaps. First, we introduce a Bias Compensation module to coarsely align initial speech representations across languages. Second, we impose token-level OT constraints on a Q-Former using parallel speech pairs to establish fine-grained consistency of representations. Then, we apply a layer scheduling strategy to focus OT constraints on the most semantically beneficial layers. Experiments on the FLEURS dataset show that our method achieves SOTA performance, with +0.93 BLEU on average over five common languages and +5.05 BLEU on zero-shot languages, using only 10 hours of parallel speech per source language. † † address: 1 Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin University, Tianjin, China 2 Nanyang Technological University, Singapore 3 Huiyan Technology Company, Ltd., Tianjin, China 4 Chinese Academy of Sciences, Guangdong, China 5 Tibet University, Lhasa, China 1 Introduction The rapid advances of large language models (LLMs) [ 3 , 24 , 18 ] in natural language processing tasks [ 2 ] are driving a paradigm shift within the speech community [ 15 ] . SpeechLLMs have consequently emerged, marking a transition from the conventional encoder-decoder pipeline to a unified, language-model-centric framework [ 22 , 5 , 26 ] . By mapping speech representation into a textual embedding space, these models support reasoning and generation directly from speech [ 15 ] . Building on this foundation, Speech-to-Text Translation (S2TT) has emerged as one of the most promising applications of SpeechLLMs [ 9 , 5 ] . The primary objective of S2TT is to efficiently convert spoken input in a source language into textual output in a target language without intermediate transcription [ 20 ] . This capability plays a vital role in real-world multilingual scenarios [ 20 ] . Consequently, such a system should be capable of handling diverse language pairs. Figure 1: (a): Language-specific clusters before alignment. (b): Utterances with the identical semantics converge toward a shared representation (pivot language). Figure 2: Model Architecture of Our Cross-Lingual Speech Alignment Framework Yet fulfilling this requirement remains challenging because state-of-the-art S2TT models achieve high accuracy on high-resource pairs but lag on low-resource pairs [ 20 ] . Although this performance bias is partially attributable to skewed training data distributions, the deeper underlying cause lies in insufficient cross-lingual comparability of speech representations [ 1 , 13 ] . As illustrated in Fig. 1 . (a), we select the state-of-the-art multilingual encoder Whisper [ 19 ] and visualize its speech representations, which form language-specific clusters: samples from the identical language coalesce rather than being grouped by shared semantic content. This prevents the decoder from effectively reusing learned translation mappings, thus amplifying the performance gap. Cross-lingual alignment is therefore the key to mitigating this bias. As illustrated in Fig. 1 . (b), this alignment encourages the model to induce a cross-lingually shared representation: an abstract ‘pivot language’. Nevertheless, current alignment research in the speech domain is confined to speech-to-text mappings. By bridging the modality gap between speech and text, these methods facilitate knowledge transfer from machine translation (MT) decoders [ 6 , 14 , 12 ] . While such a strategy highlights the importance of cross-modal modeling, it implicitly carries two limitations: First, alignment is typically constrained to the speech-to-text direction, where the learning objective is defined by textual supervision [ 14 , 12 ] . As a result, the speech encoder is forced to follow text-driven constraints, often compressing fine-grained acoustic details not represented in text. Second, this paradigm treats each source language independently, neglecting the underlying structural similarities and transferable representations shared across languages [ 23 ] . This isolation restricts cross-lingual knowledge transfer and reduces the capacity for generalization. Motivated by these observations, we propose POTSA (Parallel Optimal Transport for Speech Alignment), a novel alignment framework based on cross-lingual parallel speech pairs and Optimal Transport (OT) [ 16 ] to explicitly strengthen the consistency of cross-lingual representations for SpeechLLM in S2TT. The framework trains only the Q-Former [ 10 ] to project encoder outputs cross-lingually, while freezing both the encoder and LLM. First, we introduce a Bias Compensation module to mitigate language-specific shifts at the encoder output, narrowing cross-lingual gaps and preparing for fine-grained alignment within the speech representations. Subsequently, we leverage cross-lingual parallel speech pairs with identical semantic content and impose token-level OT constraints within the Q-Former. This constraint achieves a finer-grained alignment of source speech representations. Finally, we adopt layer-wise contributions and an online reward-guided scheduling strategy that applies OT constraints to the most effective layers. Experiments on the FLEURS dataset show state-of-the-art performance, with +0.93 BLEU across five common languages and +5.05 BLEU on zero-shot languages using only 10 hours of parallel speech per source language. Our code is publicly available at 1 1 1 https://github.com/Sslnon/POTSA . 2 PROPOSED METHOD Fig. 2 provides an overview of the SpeechLLM architecture adopted in this study. Our goal is to improve cross-lingual consistency of source speech representations in S2TT. We integrate an alignment framework into the model, as highlighted by the dashed box. 2.1 Bias Compensation Module We propose a Bias Compensation Module to coarsely align source speech representations and enforce cross-lingual consistency. We assume that each encoder output can be additively decomposed into a language-neutral component and a language-specific bias term, i.e., H^{(i)}_{x}=\tilde{H}^{(i)}_{x}+b_{x} [ 11 , 21 ] . Here, \mathbf{H}_{x}^{(i)} denotes the encoder representation of the i -th utterance in language x , whereas \mathbf{b}_{x} captures language-dependent bias shared by all utterances of that language. In practice, we estimate \mathbf{b}_{x} by averaging sentence-level representations after temporal pooling: \small\mathbf{b}_{x}=\frac{1}{N_{x}}\sum_{n=1}^{N_{x}}\text{Pool}\bigl(\mathbf{H}_{x}^{(n)}\bigr). (1) Then, subtract this estimated bias from each utterance: \small\tilde{\mathbf{H}}_{x}^{(n)}=\mathbf{H}_{x}^{(n)}-\mathbf{b}_{x}, (2) where N_{x} is the number of utterances in language x . This coarse alignment step reduces global language-specific discrepancies and establishes a more comparable representation space, thereby facilitating the subsequent fine-grained token-level alignment. 2.2 Token-wise OT Constraints for Q-Former After coarse alignment by removing language-specific global bias, we propose a fine-grained alignment strategy based on Optimal Transport (OT) and cross-lingual parallel speech pairs. Since parallel speech pairs convey identical semantic content, we use them as alignment anchors only during training. Only a limited quantity of such data is sufficient to learn robust cross-lingual representations. For each pair of parallel utterances (x_{s},x_{t}) , the projection layer produces two token sequences of embeddings, h_{s}=\{u_{i}\}_{i=1}^{n} and h_{t}=\{v_{j}\}_{j=1}^{m} . To reinforce consistency of cross-lingual semantic representations, we measure and minimize the discrepancy using the Sinkhorn distance—an entropy-regularized [ 16 ] approximation of the OT objective. Let Z\in\mathbb{R}_{+}^{n\times m} denote the transport plan under soft marginal constraints (typically uniform over valid tokens), and let c(u_{i},v_{j}) denote the ground cost (e.g., cosine distance). The loss is: \small\mathcal{L}_{\text{ot}}=\min_{Z\geq 0}\sum_{i=1}^{n}\sum_{j=1}^{m}Z_{ij}\,c(u_{i},v_{j})+\varepsilon\sum_{i=1}^{n}\sum_{j=1}^{m}Z_{ij}\log Z_{ij}, (3) where \varepsilon controls the smoothness (or sparsity) of the transport plan.