Zero-Shot Speech LLMs for Multi-Aspect Evaluation of L2 Speech: Challenges and Opportunities
Aditya Kamlesh, Cristian, Catia, Helmer · 2026
Read Zero-Shot Speech LLMs for Multi-Aspect Evaluation of L2 Speech: Challenges and Opportunities by Aditya Kamlesh, Cristian, Catia, Helmer (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16230.txt.
Introduction
Aditya Kamlesh and coauthors test speech models on scores for learned tongue speech. Their 2026 preprint deals with scores for speech in a second tongue. Good scores give fair feed back and track growth. Speech must be judged on flow, tune and full sense at once. That joint score task is hard for code tools.
The team tests speech large models with no task train. Such zero shot tests show what the models can do at once. They weigh scores across all parts of speech grade. Gaps show where fresh train or cues are needed. The work maps a path to fair speech scores.
Catalog record
| CREATOR | Aditya Kamlesh, Cristian, Catia, Helmer |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-16230
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Zero-Shot Speech LLMs for Multi-Aspect Evaluation of L2 Speech: Challenges and Opportunities Aditya Kamlesh, Cristian, Catia, Helmer Abstract An accurate assessment of L2 English pronunciation is crucial for language learning, as it provides personalized feedback and ensures a fair evaluation of individual progress. However, automated scoring remains challenging due to the complexity of sentence-level fluency, prosody, and completeness. This paper evaluates the zero-shot performance of Qwen2-Audio-7B-Instruct, an instruction-tuned speech-LLM, on 5,000 Speechocean762 utterances. The model generates rubric-aligned scores for accuracy, fluency, prosody, and completeness, showing strong agreement with human ratings within ±2 tolerance, especially for high-quality speech. However, it tends to overpredict low-quality speech scores and lacks precision in error detection. These findings demonstrate the strong potential of speech LLMs in scalable pronunciation assessment and suggest future improvements through enhanced prompting, calibration, and phonetic integration to advance Computer-Assisted Pronunciation Training. Parikh Tejedor-Garcia Cucchiarini Strik Radboud Universitythe Netherlands keywords L2 pronunciation assessment, Multimodal speech processing, Speech Large language models (LLMs), Zero-shot speech evaluation, Computer-Assisted Pronunciation Training (CAPT) † † email: [email protected], [email protected], [email protected], [email protected] 1 Introduction Globalization has increased cross-border movement for work, education, and other opportunities, making local language acquisition vital for integration, career growth, and well-being [ 22 ] . Beyond vocabulary and grammar, clear pronunciation is crucial for intelligibility, confidence, and meaningful interaction [ 24 , 8 ] . Poor pronunciation can hinder communication and create barriers in academic and professional contexts [ 25 ] . However, the pronunciation of second-language (L2) is difficult to master due to differences from the mother tongue (L1) and a lack of tools for personalized feedback. Computer-Assisted Language Learning (CALL), particularly Computer-Assisted Pronunciation Training (CAPT), has become essential in language education in the past years [ 15 ] . These tools offer scalable, cost-effective, and consistent pronunciation feedback, addressing the limitations of human evaluation. Traditional CAPT systems use phoneme-level scoring methods like Goodness of Pronunciation (GOP), which assess individual sounds for correctness [ 18 ] . However, they often fail to address sentence-level fluency, prosody, and completeness, and typically do not provide corrective feedback to learners [ 6 ] . Recent advancements in Large Language Models (LLMs) have significantly transformed the landscape of natural language processing. While LLMs have demonstrated outstanding performance in text-based tasks [ 1 ] , there is growing interest in extending their capabilities to handle other modalities, such as speech [ 14 , 9 ] . Typically, these speech LLMs integrate pre-trained speech encoders with text-based LLMs via projection layers that align feature dimensions across modalities [ 20 ] . While LLMs are widely used in writing assessment [ 11 , 21 ] , their application to spoken language assessment is still emerging. Previous work, such as [ 16 ] , explored text-only models like ChatGPT for pronunciation feedback. To overcome the limitations of text-only models, multimodal LLMs like GPT-4o [ 7 ] have been developed to process raw audio input. These models assess pronunciation across dimensions such as accuracy, fluency, prosody, and sentence completeness, and generate context-aware feedback. Recent studies [ 20 , 19 ] introduced Audio-Text Prompt LLMs using wav2vec2 encoders and LLaMA2 decoders, enhancing performance by embedding mispronunciations into prompts. Models like Qwen-Audio [ 4 , 3 ] , extensively trained on both supervised and unsupervised speech data at a large scale, followed by instruction tuning, have shown good results in speech tasks such as ASR, translation, audio captioning, and spoken question answering. [ 10 ] explored the potential of speech LLM for L2 oral proficiency assessment by evaluating zero-shot capabilities and proposing fine-tuning strategies, ultimately demonstrating that adapted models significantly outperform traditional baselines. Building on recent advances in audio-text prompt-based language models, we examine the potential of the Qwen-Audio-Instruct model for spoken language assessment. We adopt the term multi-aspect to refer to the simultaneous evaluation of multiple dimensions of speech performance, i.e., accuracy, fluency, prosody, and sentence completeness, rather than focusing solely on phoneme-level correctness [ 5 ] . To the best of our knowledge, this is the first work that evaluates an instruction-tuned, speech LLM in a zero-shot setting for multi-aspect pronunciation assessment using rubric-based scoring. The model combines a powerful audio encoder with an instruction-tuned decoder, making it a strong candidate for the holistic evaluation of L2 speech. In this work, we explicitly ask the following research question: To what extent can the Qwen-Audio-Instruct model perform multi-aspect spoken language assessment, covering accuracy, fluency, prosody, and sentence completeness, and generate rubric-aligned scores in a zero-shot setting? 2 Methodology We evaluate the zero-shot performance of the Qwen2-Audio-7B-Instruct model 1 1 1 https://huggingface.co/Qwen/Qwen2-Audio-7B-Instruct for multi-aspect spoken language assessment. This instruction-tuned, multimodal large language model (LLM) integrates a Whisper-based speech encoder and a transformer-based text decoder, enabling direct audio-text interaction without task-specific fine-tuning. We adopt a zero-shot setting to assess the model’s generalization ability—specifically, its capacity to follow rubric-based instructions without prior adaptation. Zero-shot evaluation has emerged as a compelling paradigm for assessing the transfer capabilities of instruction-tuned models, particularly in low-resource or domain-agnostic scenarios [ 17 , 13 ] . 2.1 Dataset We use the Speechocean762 dataset [ 23 ] , a publicly available corpus designed for L2 pronunciation assessment. It comprises 5,000 English utterances from 250 Mandarin-speaking learners, with balanced representation across children and adults. Each utterance is manually rated by five expert annotators at the sentence, word, and phoneme levels. For this study, we focus on the sentence-level annotations, which include four rubrics: accuracy , fluency , prosody , and completeness . These scores are provided on a 0–10 scale and are suitable for testing multi-aspect scoring in a zero-shot setting. 2.2 Model Architecture The Qwen2-Audio model comprises two primary components: • Audio Encoder: Based on Whisper-large-v3 [ 12 ] , it converts raw waveforms into mel-spectrograms (16kHz) and produces downsampled audio embeddings aligned with the language model’s input space. • Text Decoder: Built on the Qwen-7B architecture [ 2 ] , a decoder-only transformer trained on trillions of tokens. It receives the audio embeddings and instruction prompts to generate structured outputs. Instruction-following behavior is enhanced through supervised fine-tuning and Direct Preference Optimization (DPO). 2.3 Prompting Strategy Each learner’s utterance is paired with a single, multi-modal prompt containing both the raw audio and the expected reference text. 2 2 2 https://github.com/Aditya3107/sentence-scoring-qwen This strategy allows the model to simultaneously process and assess pronunciation across rubrics, thereby reflecting a realistic, integrated evaluation scenario. In particular, the model is instructed to evaluate the speaker’s pronunciation holistically using four sentence-level rubrics: accuracy , fluency , prosody , and sentence completeness . These rubrics are adapted from the original Speechocean762 guidelines [ 23 , Section 3.1, Table 1] , and refined using detailed descriptions to enhance the model’s interpretability. The prompt explicitly defines each rubric with a scoring scale from 0 to 10, along with criterion-referenced descriptions for each level. For instance, accuracy is defined in terms of the number and severity of mispronunciations, while fluency considers speech coherence, pause distribution, and hesitations. Prosody focuses on the use of intonation, rhythm, and stress patterns to convey natural speech flow. Completeness refers to the extent to which the entire target sentence was spoken, regardless of pronunciation correctness or intelligibility. The prompt is phrased to reflect a real-world assessment context