Fine-Tuning Large Multimodal Models for Automatic Pronunciation Assessment
Ke Wang, Wenning Wei, Yan Deng, Lei He, Sheng Zhao · 2025
Read Fine-Tuning Large Multimodal Models for Automatic Pronunciation Assessment by Ke Wang, Wenning Wei, Yan Deng, Lei He, Sheng Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15701.txt.
Introduction
Ke Wang leads a team that grades pronunciation with large models that take sound and text. Such grading aids language learning apps. It must score at the level of sounds, words, and full sentences, across several traits. Whether big multimodal models can do this fine work was unclear.
They tune such a model on a public data set plus a private one. Tuning beats no tuning by far and rivals public and commercial systems on single level tasks. Word and sentence level scores are strong. Sound level grading stays hard. Two score types split in an odd way, and the team argue the rank based one better shows true order. Future work targets finer modeling.
Catalog record
| CREATOR | Ke Wang, Wenning Wei, Yan Deng, Lei He, Sheng Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-15701
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Fine-Tuning Large Multimodal Models for Automatic Pronunciation Assessment Ke Wang, Wenning Wei, Yan Deng, Lei He, Sheng Zhao Abstract Automatic Pronunciation Assessment (APA) is critical for Computer-Assisted Language Learning (CALL), requiring evaluation across multiple granularities and aspects. Large Multimodal Models (LMMs) present new opportunities for APA, but their effectiveness in fine-grained assessment remains uncertain. This work investigates fine-tuning LMMs for APA using the Speechocean762 dataset and a private corpus. Fine-tuning significantly outperforms zero-shot settings and achieves competitive results on single-granularity tasks compared to public and commercial systems. The model performs well at word and sentence levels, while phoneme-level assessment remains challenging. We also observe that the Pearson Correlation Coefficient (PCC) reaches 0.9, whereas Spearman’s rank Correlation Coefficient (SCC) remains around 0.6, suggesting that SCC better reflects ordinal consistency. These findings highlight both the promise and limitations of LMMs for APA and point to future work on fine-grained modeling and rank-aware evaluation. † † address: Microsoft, Beijing, China 1 Introduction Computer-Assisted Language Learning (CALL) provides an economical and scalable approach for language learners to improve multiple aspects of spoken language proficiency, including fluency, accuracy, prosody, intelligibility, grammar, and vocabulary [ 1 ] . With increasing globalization and closer international collaboration, proficiency in foreign languages and strong oral communication skills have become essential. In this context, CALL systems play a pivotal role by delivering immediate and personalized feedback to support effective communication across diverse linguistic environments. A typical CALL system includes components such as Mispronunciation Detection and Diagnosis (MDD), Automatic Pronunciation Assessment (APA), and Automated Essay Scoring (AES) [ 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ] . MDD focuses on identifying phone-level errors by aligning recognized phone sequences with canonical references. Common approaches include the Goodness of Pronunciation (GOP) method [ 10 , 11 ] and Extended Recognition Network (ERN) [ 12 , 13 ] , as well as recent end-to-end models leveraging CNN-RNN-CTC architectures [ 14 ] and self-supervised learning with wav2vec 2.0 [ 15 ] . While these methods provide detailed feedback, they primarily emphasize accuracy and often lack prosody and fluency evaluation [ 6 , 16 ] . In contrast, APA aims at providing a holistic evaluation across multiple aspects (accuracy, fluency, prosody, completeness, stress and total) at different granularities, including phoneme, word, and sentence levels. Traditional APA systems primarily rely on ASR-derived features and statistical models, which often struggle to capture nuanced prosodic and fluency patterns. Recent research has advanced toward multi-granularity and multi-aspect frameworks [ 5 , 6 , 7 , 17 , 18 ] and integrated MDD–APA models [ 19 , 20 ] , leveraging modern neural architectures such as Transformers and Mamba to enhance assessment accuracy and robustness. Recent advances in Large Language Models (LLMs) and Large Multimodal Models (LMMs) have enabled strong language understanding and multimodal reasoning, driving their adoption in CALL tasks. Wang et al. [ 16 ] explored LLMs for phrase break evaluation, finding performance gaps compared to their proposed method. Other studies prompted LLMs with audio and text for MDD [ 3 ] and introduced multimodal LLMs for sentence-level accuracy and fluency scoring [ 8 ] , achieving competitive results. Recent work [ 9 , 21 ] also examined LLMs for APA and feedback generation. With LMMs, researchers have begun fine-tuning models for L2 oral proficiency assessment without additional audio adapters [ 22 ] , marking progress in overall score prediction. Despite these advances, LMMs’ ability to handle multi-granularity and multi-aspect APA remains underexplored. APA requires not only content understanding but also extraction of speech features such as accuracy, fluency, and prosody [ 9 ] . Nearly at the same time with this study, LoRA-based [ 23 ] fine-tuning has been applied to Phi-4 multimodal models for APA and MDD [ 24 ] , but prior work focused only on sentence-level scoring. Moreover, the scarcity of APA-specific data limits LMM pretraining in this domain. To address these gaps, we investigate fine-tuning strategies for APA across multiple granularities and aspects using Qwen2-Audio-7B-Instruct [ 25 ] , hereafter referred to as Qwen2-Audio, chosen for its strong text–audio capabilities and instruction-following design for speech-related tasks. Our contributions are threefold: • We systematically evaluate the performance of fine-tuned LMMs on APA tasks in both multi-granularity and multi-aspect settings, as well as single-granularity and single-aspect settings. • We identify and analyze the limitations of fine-tuned LMMs in phoneme-level assessment. • We provide insights into evaluation metrics, highlighting the importance of rank-based measures such as Spearman’s rank Correlation Coefficient (SCC) for APA. 2 Methodology Qwen2-Audio is capable of processing various types of audio inputs and operating in two distinct modes: audio analysis mode, which focuses on extracting and interpreting acoustic features, and voice chat mode, which generates direct textual responses based on spoken instructions. Due to its open-source availability and strong performance in audio-language tasks, we adopt Qwen2-Audio as the backbone for our APA framework. Unlike previous approaches that require an external audio encoder [ 3 , 8 ] , Qwen2-Audio natively supports both audio and text inputs, significantly simplifying the pipeline and reducing system complexity for future research. The overall structure of the proposed framework is illustrated in Fig. 1 , and we employ LoRA for parameter-efficient fine-tuning of the LMM. Figure 1 : Overall structure of the proposed method. 2.1 Prompts Prompt engineering plays a critical role in adapting LMMs for APA tasks. We design multiple prompts tailored to different granularities and aspects of assessment. Below is an example of the comprehensive prompt used for training and evaluation, following the annotation guidelines: You are a pronunciation evaluation teacher who analyzes speech at sentence, word and phone levels. At the sentence level, the evaluation includes accuracy, fluency, prosody, completeness, and a total score. Accuracy reflects phone-level pronunciation, accent, and clarity; fluency considers pauses, repetitions, and stammering; prosody assesses intonation, speed, and rhythm; and completeness indicates the percentage of words from the target text that were actually pronounced correctly. The total score is a comprehensive measure that reflects all aspects of speech. At the word level, the evaluation includes accuracy, stress, and a total score. Accuracy again considers phone pronunciation, accent, and clarity, while the stress score reflects the correctness of stress placement, where a score of 10 indicates correct stress and 5 indicates incorrect stress. At phone level, we assess the goodness of each phoneme with in the words. All scores range from 0 to 10, with 0 representing the poorest pronunciation and 10 representing the best, except for the word-level stress score, which is either 5 or 10.\n Please evaluate the provided audio at the sentence, word, and phone levels using the given reference text and phone sequence.\n Reference text: { REFERENCE TEXT } .\n Reference phone sequence: { REFERENCE PHONE SEQUENCE } .\n In the phone sequence, ‘-’ indicates word boundaries. The final output format should be structured as follows:\n Sentence Scores: { Acc } { Flu } { Pro } { Com } { Tot } \n Word Scores: { W1 } / { Acc } / { Str } / { Tot } { W2 } / { Acc } / { Str } / { Tot } …\n Phone Scores: { P1