Cross-modal Knowledge Transfer Learning as Graph Matching Based on Optimal Transport for ASR
2025
Read Cross-modal Knowledge Transfer Learning as Graph Matching Based on Optimal Transport for ASR (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13079.txt.
Introduction
Xugang Lu and three co-authors improve speech recognition by borrowing knowledge from text models in a 2025 paper. A strong language model knows grammar and words, and that knowledge can coach a speech system. The hard part is aligning text features with sound features, since the two live in different shapes.
Older methods lined the two up as loose sets of points, ignoring order. The new method treats each sequence as a graph, with points for features and edges for time order. A combined distance then matches both the points and the wiring between them. Theory shows the older approach is a special case of this one. Tests on Mandarin speech recognition showed clear gains. The authors say respecting structure is what lets knowledge cross from reading to listening.
Catalog record
| YEAR | 2025 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13079
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cross-modal Knowledge Transfer Learning as Graph Matching Based on Optimal Transport for ASR Thanks: *Acknowledgment: The work is partially supported by JSPS KAKENHI No. 24K15004. Abstract Transferring linguistic knowledge from a pretrained language model (PLM) to acoustic feature learning has proven effective in enhancing end-to-end automatic speech recognition (E2E-ASR). However, aligning representations between linguistic and acoustic modalities remains a challenge due to inherent modality gaps. Optimal transport (OT) has shown promise in mitigating these gaps by minimizing the Wasserstein distance (WD) between linguistic and acoustic feature distributions. However, previous OT-based methods overlook structural relationships, treating feature vectors as unordered sets. To address this, we propose Graph Matching Optimal Transport (GM-OT), which models linguistic and acoustic sequences as structured graphs. Nodes represent feature embeddings, while edges capture temporal and sequential relationships. GM-OT minimizes both WD (between nodes) and Gromov-Wasserstein distance (GWD) (between edges), leading to a fused Gromov-Wasserstein distance (FGWD) formulation. This enables structured alignment and more efficient knowledge transfer compared to existing OT-based approaches. Theoretical analysis further shows that prior OT-based method in linguistic knowledge transfer can be viewed as a special case within our GM-OT framework. We evaluate GM-OT on Mandarin ASR using a CTC-based E2E-ASR system with a PLM for knowledge transfer. Experimental results demonstrate significant performance gains over state-of-the-art models, validating the effectiveness of our approach. keywords optimal transport, pretrained languaue model (PLM), Gromov-Wasserstein distance, transport coupling † † address: Xugang Lu 1∗ , Peng Shen 1 , Yu Tsao 2 , Hisashi Kawai 1 1. National Institute of Information and Communications Technology, Japan 2. Research Center for Information Technology Innovation, Academia Sinica, Taiwan † † email: [email protected] 1 Introduction The pretrained language models (PLMs) learned from large text corpora encodes rich linguistic knowledge, and has greatly advanced the performance of automatic speech recognition (ASR) with end-to-end (E2E) acoustic models [ 1 , 2 , 3 , 4 , 5 ] . While enhancing ASR in decoding, typically as external language models (LMs) for tasks like beam search and rescoring [ 6 , 7 ] , external LMs introduce computational inefficiencies and hinder parallel decoding. A key challenge is transferring linguistic knowledge into the acoustic encoding stage, enabling ASR without external LM dependencies. This work explores knowledge transfer from PLMs to a temporal connectionist temporal classification (CTC)-based ASR [ 8 ] . While some E2E-ASR methods integrate linguistic information [ 9 , 10 ] , leveraging PLMs like BERT [ 11 ] has shown promise [ 12 , 13 , 14 , 15 , 17 , 18 , 19 , 20 , 21 , 22 , 23 ] . In their models, cross-attention based alignment model in transformer-based decoding [ 24 ] , or Continuous Integrate-and-Fire (CIF) neural network-based alignment model [ 19 , 20 ] are applied in integrating linguistic knowledge with acoustic models. However, due to modality gaps, aligning acoustic and linguistic feature representations still remains a challenge. Optimal transport (OT) provides a natural framework for cross-modal alignment by minimizing feature distributional discrepancies [ 25 ] . OT has been applied to domain adaptation [ 26 ] , spoken language recognition, speech enhancement [ 27 , 30 , 31 ] , and speech understanding [ 32 , 33 , 34 ] . Recent studies explore OT-based knowledge transfer learning for ASR [ 28 , 29 ] , but most treat feature representations as unordered sets, ignoring the structured nature of speech. In reality, acoustic features exhibit temporal coherence, and linguistic features follow a sequential order. We hypothesize that incorporating these structural properties into OT can improve alignment accuracy and efficiency in linguistic knowledge transfer. To address this, we introduce Graph Matching Optimal Transport (GM-OT), framing cross-modal alignment as a graph-matching problem. Each utterance is represented as two graphs, one in acoustic space and one in linguistic space with nodes as feature embeddings and edges capturing temporal and sequential structures. Effective alignment requires matching both nodes and edges, ensuring feature similarity and structural coherence, ultimately improving E2E-ASR performance. Our main contributions are as follows: 1. We propose a novel GM-OT framework for linguistic knowledge transfer in E2E-ASR, leveraging graph-based optimal transport for structured feature alignment. 2. We provide theoretical insights, showing that existing OT-based alignment and transfer learning methods can be derived as special cases of our GM-OT framework. 3. We conduct experiments on a CTC-based E2E-ASR system incorporating a PLM for knowledge transfer. Our results demonstrate significant performance gains in linguistic knowledge transfer learning for E2E-ASR. 2 Proposed method The proposed model framework is illustrated in Fig. 1 . In this framework, two encoders, i.e., acoustic encoder and linguistic encoder are used for exploring acoustic and linguistic feature representations. Between the two encoders, there is one ‘Adapter’ module to align acoustic feature dimension with the linguistic one (gray blocks in Fig. 1 ). The main difference between our work and other studies [ 28 ] is that our work introduces a GM-OT alignment module for matching these representations. Before detailing GM-OT, we briefly explain the feature exploration blocks. Figure 1: The proposed cross-modal alignment and knowledge transfer model for ASR based on GM-OT. 2.1 Feature exploration in acoustic and linguistic encoders In acoustic encoder, convolution-based down-sampling is applied in order to reduce the duration lengths of acoustic sequence (a down-sampling rate of 4 was used in this paper). After down-sampling, a positional encoding is applied in order to keep temporal position sensitivity in acoustic encoding. The representation from acoustic encoder is obtained as: {\bf H}_{{\rm enc}}={\rm AcousticEncoder}\left({{\bf X}}\right)\in\mathbb{R}^{l_{a}\times d_{a}}, (1) where l_{a} and d_{a} are temporal length and dimension of the acoustic feature vectors, respectively. A linear projection termed ‘FC2’ in the ‘Adapter’ module is utilized to perform a feature dimension matching transform as: {\bf H}^{\rm A}={\rm FC}_{\rm 2}\left({{\bf H}_{{\rm enc}}}\right)\in\mathbb{R}^{l_{a}\times d_{t}}, (2) where d_{t} corresponds to linguistic feature dimension. The context-dependent linguistic feature representation is explored from a linguistic encoder. The process is formulated as: \begin{array}[]{l}{\bf Z}_{0}=\left[{{\rm CLS,}{\bf y}_{{\rm token}},{\rm SEP}}\right]\\ {\bf Z}^{\rm L}={\rm LinguisticEncoder}\left({{\bf Z}_{0}}\right)\in\mathbb{R}^{l_{t}\times d_{t}},\\ \end{array} (3) where {\bf y}_{{\rm token}} are word piece-based tokens, e.g., tokens by tokenization process of bidirectional encoder representation from transformers (BERT) [ 11 ] . Token symbols ‘CLS’ and ‘SEP’ represent the start and end of an input token sequence. In Eq. ( 3 ), l_{t} denotes the sequence length, and d_{t} represents feature dimension of text encoding representation. 2.2 Feature alignment based on Optimal Transport After extracting acoustic and linguistic features, alignment is needed for efficient knowledge transfer. OT maps one probability distribution to another with minimal transport cost [ 25 ] and has been used for representation alignment in ASR transfer learning [ 28 ] . However, their usage of OT treats feature vectors as an unordered set, ignoring speech and linguistic structures. We propose a new perspective, modeling each utterance as graphs in acoustic and linguistic spaces, where feature vectors are nodes and their relationships form edges. Fig. 2 illustrates the difference between vanilla OT and our graph-matching-based OT (GM-OT) for alignment. Figure 2: Feature alignment between acoustic and linguistic modalities, (a) OT on unordered set, (b) OT on set with pair-wised relevance, (c) OT on set with temporal consistent topology structure. In Fig. 2 (a), traditional OT uses Wasserstein distance (WD) on