Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysis

Tianyi, Hongjie, Wang, Lv, Jian, Li, Zhennan, Yongxiang, Xie · 2025

Read Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysis by Tianyi, Hongjie, Wang, Lv, Jian, Li, Zhennan, Yongxiang, Xie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21138.txt.

Introduction

Speech recognition still falters on Chinese dialects, where training speech is scarce beside the flood available for Mandarin. Tianyi Xu and co-authors test in 2025 whether self supervised pretraining plus a large language model can close that gap. They pretrain on 300,000 hours of unlabeled dialect speech, then align the model on 40,000 hours of labeled data.

They vary the connector pieces and the language model paired with the system, checking Mandarin, dialect, and accented speech alike. The recipe set state of the art marks on several dialect test sets. The authors promise to open source the work. They frame the result as a route for other low resource varieties of speech.

Catalog record

CREATORTianyi, Hongjie, Wang, Lv, Jian, Li, Zhennan, Yongxiang, Xie
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21138

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Leveraging LLM and Self-Supervised Training Models for Speech Recognition in Chinese Dialects: A Comparative Analysis Tianyi, Hongjie, Wang, Lv, Jian, Li, Zhennan, Yongxiang, Xie Abstract Large-scale training corpora have significantly improved the performance of ASR models. Unfortunately, due to the relative scarcity of data, Chinese accents and dialects remain a challenge for most ASR models. Recent advancements in self-supervised learning have shown that self-supervised pretraining, combined with large language models (LLM), can effectively enhance ASR performance in low-resource scenarios. We aim to investigate the effectiveness of this paradigm for Chinese dialects. Specifically, we pre-train a Data2vec2 model on 300,000 hours of unlabeled dialect and accented speech data and do alignment training on a supervised dataset of 40,000 hours. Then, we systematically examine the impact of various projectors and LLMs on Mandarin, dialect, and accented speech recognition performance under this paradigm. Our method achieved SOTA results on multiple dialect datasets, including Kespeech. We will open-source our work to promote reproducible research. 1 1 1 Work done during internship at China Telecom. Xu Chen Qing Hang Kang Jie Lin Li Lei China TelecomChina Northwestern Polytechnical UniversityChina keywords speech recognition, LLM, speech foundation model † † email: [email protected]; [email protected] 1 Introduction Large-scale training corpora have significantly improved the performance of automatic speech recognition (ASR) models [ 1 , 2 , 3 , 4 ] . However, labeled dialect data remains prohibitively expensive, making it challenging to obtain in large quantities. As a result, most pre-trained models in Chinese speech processing communities, such as WeNet [ 5 ] , K2 [ 6 ] , and FunASR [ 7 ] , exhibit limited capabilities in dialect recognition. Fortunately, advances in self-supervised learning have demonstrated the potential of large-scale training to further improve model performance. Self-supervised large language models (LLM) [ 8 , 9 , 10 , 11 ] have shown exceptional capabilities to understand and generate human language, and self-supervised speech models such as Wav2Vec2 [ 12 ] , HuBERT [ 13 ] and Data2Vec2 [ 14 ] have demonstrated strong proficiency in modeling speech. Given that ASR traditionally relies on both acoustic and language modeling, the advantages of self-supervised pre-trained acoustic models and LLMs—particularly their scalability in terms of training data and model size—present promising opportunities for low-resource ASR systems. Research efforts to integrate LLMs with ASR systems generally follow two primary approaches. The first is the cascaded approach [ 15 , 16 , 17 ] , where LLMs are employed to refine the n-best hypotheses generated by ASR systems. Among studies adopting this approach, MMGER [ 18 ] stands out as a closely related work, achieving the previous SOTA performance in Chinese accents on the Kespeech [ 19 ] dataset. However, this approach has a notable limitation: for complex cases, the n-best hypotheses may fail to provide sufficiently informative input to the LLM, making it difficult for the model to infer correct results from erroneous predictions. This limitation can sometimes result in semantic distortions during error correction. The second approach involves training audio-text cross-modal LLMs [ 20 , 21 , 22 ] , where a speech encoder processes audio signals to generate embeddings that are subsequently fed into an LLM for decoding. This method aims to integrate acoustic features more effectively with linguistic context to enhance ASR accuracy. Specifically, SALMONN [ 20 ] employs Whisper [ 1 ] to extract semantic content while utilizing BEATs [ 23 ] to enable comprehensive perception of speech, music, and sound events. Qwen-Audio and Qwen2-Audio [ 22 , 24 ] use Whisper as an encoder and optimize task performance across various domains with structured task instructions. Similarly, SLAM-ASR [ 21 ] achieved SOTA results on the 960-hour English LibriSpeech dataset by training only the projector. In the context of Chinese ASR, Geng et al. [ 25 ] achieved SOTA performance on AISHELL, Test-net, and Test-meeting by leveraging HuBERT encoder and Transformer projection layers. Figure 1: Overall model structure. Our study extends these investigations by exploring the potential of the speech foundation encoder with the LLM-Decoder paradigm on large-scale Chinese dialect datasets. Specifically, we utilize two datasets: an unlabeled dataset consisting of approximately 300,000 hours of dialect and accented speech and a labeled dataset comprising 40,000 hours. Using these resources, we systematically investigate the impact of various speech encoders, projectors, and LLMs on the ASR performance of Chinese dialects. Our experimental results reveal several important findings: • First, among speech encoders, Data2Vec2 is a good speech encoder for large language models and can have competitive performance when downsampled to 25Hz. Combined with the 4-time downsampling from the projector, it can achieve competitive performance at 6.25 Hz framerate for the LLM, greatly reducing the computational overhead. • Second, fully connected layers outperform other projectors, particularly during the early stages of training. Notably, differences in the effectiveness of projectors are most prominent during the first training stage but diminish as the parameters of the pre-trained models are progressively unfrozen in subsequent stages. • Third, a multi-stage training strategy proves highly effective in enhancing model performance, with the addition of a fourth training stage, building upon conventional three-stage approaches, yields further improvements. Compared to previous work [ 21 , 25 ] , our approach employs Data2Vec2 as the speech encoder and incorporates convolutional layers as the projector. Combined with a four-stage training strategy, the proposed method achieves SOTA performance on multiple Chinese dialect and accent test sets, including Kespeech (with CER of 6.48% vs. 7.52%), even with a small LLM (0.5B) and a low frame rate (6.25Hz). To promote reproducibility, we will release all associated training, inference, and evaluation scripts, as well as pre-trained models and training logs. 2 Method As illustrated in Figure 1 , the model architecture consists of an audio encoder and a LLM. For each sample during training, we define the text prompt (e.g., ”Transcribe the following speech”), the speech utterance, and the corresponding transcription as P , S and T , respectively. The text prompt and transcription are tokenized using the LLM’s tokenizer and embedded into feature vectors E_{p} and E_{t} as: \displaystyle E_{p} \displaystyle=\text{Embedding}(\text{Tokenizer}(P)), (1) \displaystyle E_{t} \displaystyle=\text{Embedding}(\text{Tokenizer}(T)). (2) For the input speech S , features are first extracted using the speech encoder, producing the encoder output H_{s} , represented as: H_{s}=\text{Encoder}(S). (3) The encoder output H_{s} is then passed through a projector, followed by a linear layer, to generate a feature sequence E_{\text{s}} with the same dimensionality as the input to the LLM, denoted as: E_{\text{s}}=\text{Linear}(\text{Projector}(H_{s})), (4) Here, the projector preserves the dimensionality of the features output by the speech encoder, while the linear layer maps these features to the embedding space of the LLM. Finally, E_{\text{s}} , E_{p} , and E_{t} are concatenated into a unified feature sequence and fed into the LLM. The model generates the transcription Y , as the output, defined as: Y=\text{LLM}(\text{Regulation}(E_{p},E_{\text{s}},E_{t})). (5) 3 Experimental Setup 3.1 Datasets Training Set: The training data comprises a diverse set of internal dialect and accent datasets, including samples from Anhui, Gansu, Hebei, Shandong, Shanxi, Tianjin, Cantonese, Henan, Sichuan, Chongqing, Northeast China, Shaanxi, Hubei, Fujian, Guizhou, Hangzhou, Hunan, Jiangxi, Shanghai, Suzhou, Yunnan, Chongqing and Hakka. Additionally, public Mandarin datasets, such as Wenetspeech [ 26 ] , Aishell [ 27 ] , and AliMeeting,

Read the complete text