A Perception-Based L2 Speech Intelligibility Indicator: Leveraging a Rater’s Shadowing and Sequence-to-sequence Voice Conversion
Haopeng, Daisuke, Nobuaki · 2025
Read A Perception-Based L2 Speech Intelligibility Indicator: Leveraging a Rater’s Shadowing and Sequence-to-sequence Voice Conversion by Haopeng, Daisuke, Nobuaki (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24304.txt.
Introduction
Haopeng Geng, Daisuke Saito, and Nobuaki Minematsu grade how understandable second-language speech really is to a native ear. Speech tools for language learners usually score how native-like a learner sounds, which can miss true intelligibility. Their indicator borrows a native rater's shadowing, the act of repeating speech as it is heard. A sequence-to-sequence voice conversion system learns from that shadowing behavior.
An alignment mechanism and acoustic rebuild let the system mimic native listening and flag segments likely to trip up comprehension. Objective and human checks showed closer agreement with native judgments than ASR-based scores. The authors aim it at computer-assisted language learning worldwide.
Catalog record
| CREATOR | Haopeng, Daisuke, Nobuaki |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-24304
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Perception-Based L2 Speech Intelligibility Indicator: Leveraging a Rater’s Shadowing and Sequence-to-sequence Voice Conversion Haopeng, Daisuke, Nobuaki Abstract Evaluating L2 speech intelligibility is crucial for effective computer-assisted language learning (CALL). Conventional ASR-based methods often focus on native-likeness, which may fail to capture the actual intelligibility perceived by human listeners. In contrast, our work introduces a novel, perception-based L2 speech intelligibility indicator that leverages a native rater’s shadowing data within a sequence-to-sequence (seq2seq) voice conversion framework. By integrating an alignment mechanism and acoustic feature reconstruction, our approach simulates the auditory perception of native listeners, identifying segments in L2 speech that are likely to cause comprehension difficulties. Both objective and subjective evaluations indicate that our method aligns more closely with native judgments than traditional ASR-based metrics, offering a promising new direction for CALL systems in a global, multilingual contexts. Geng Saito Minematsu The University of TokyoJapan keywords computer assisted pronunciation training, speech shadowing, speech intelligibility, voice conversion † † email: {kevingenghaopeng, dsk_saito, mine}@gavo.t.u-tokyo.ac.jp 1 Introduction (a) L1 rater shadowing for L2 speech intelligibility assessment. (b) Mapping disfluency label from a rater’s shadowing to L2 speech. Figure 1 : An overview of the L1 rater’s shadowing technique for immediate L2 speech intelligibility assessment. The proposed approach is designed to identify unintelligible segments in the L2 speaker’s read-aloud utterances ( L2_{R} ). Non-native speakers are often encouraged to emulate native-like pronunciation; however, achieving a native-like accent can be time-consuming and may not efficiently improve speech intelligibility. Many studies indicate that mispronunciations or accents do not necessarily result in unintelligible speech. For example, [ 1 ] found that only 16% of word-level unintelligibility in an English spoken test can be attributed to obvious mispronunciations, and experimental evaluations in [ 2 ] show that even non-native accents, such as Swedish and German-accented English, yield higher recognition accuracy using OpenAI’s Whisper compared to British-accented English. According to [ 3 ] , approximately 1.14 billion non-native speakers account for 75% of the global English-speaking community. However, mainstream Automatic Speech Recognition (ASR) systems are typically built using native-accented corpora or the most prevalent accents, which can lead to biased performance that often results in higher error rates and misrecognition on non-native or less common accents. Additionally, recent ASR systems are specifically designed to predict—or even over-speculate on—what speakers actually said, which conflicts with the goals of computer-assisted language learning (CALL) systems [ 4 , 5 ] . From an educational perspective, language learners require feedback on whether their accents hinder communication with speakers from diverse backgrounds rather than simply striving for native-like pronunciation. Therefore, it is essential to develop methods that capture not only native-like pronunciation but also the actual intelligibility as perceived by real human listeners. How can we effectively identify unintelligible parts in L2 speech? One promising method is to measure native (L1) listeners’ responses by asking them to shadow L2 utterances—where an L1 speaker immediately repeats the L2 speech with minimal delay. This approach, as explored in previous research [ 6 , 7 , 8 ] , has proven effective at revealing where listening breakdowns occur, highlighting problematic segments in the L2 speech, and providing valuable feedback for learners. Inspired by this approach, our study aims to utilize the L1 rater’s shadowing process to build an unintelligibility indicator based on the rater’s perception. Our contributions are as follows: 1. We propose a customized speech unintelligibility indicator that leverages native rater’s shadowing data. Unlike conventional methods that focus solely on native-like pronunciation, our approach aims to capture the perceptual cues that native listeners actually use to assess intelligibility 1 1 1 Samples are available at https://secondtonumb.github.io/publication_demo/IS_2025/ . 2. Our method incorporates a state-of-the-art sequence-to-sequence (seq2seq) voice conversion framework, utilizing its alignment mechanism and acoustic feature reconstruction module to simulate the cognitive process of native listeners in identifying unintelligible segments. 3. We introduce a multi-task learning strategy that jointly optimizes speech reconstruction and disfluency detection using auxiliary loss functions. Experimental evaluations demonstrate that our approach aligns more closely with native raters’ perceptions than mainstream ASR, revealing the potential for delivering personalized feedback in CALL systems. 2 Research background 2.1 L1 rater’s shadowing Assessing L2 speech intelligibility requires fine-grained annotation, yet this process is challenging due to the need for expert-level phonetic knowledge for phoneme-level labeling. To address this challenge, [ 7 ] proposed a two-stage reverse shadowing approach to identify unintelligible segments in L2 utterances 2 2 2 In second language acquisition (SLA), shadowing usually involves learners repeating L1 speech to improve their listening skills. In our study, however, L1 raters shadow L2 speech—a process we refer to as reverse shadowing. . As illustrated in Fig 1(a) , an L1 speaker first shadows a given L2 utterance alone ( L1_{S1} ), during which listening breakdowns manifest as stuttering or inarticulate speech. Subsequently, the L1 speaker performs script-shadowing ( L1_{SS} ), where the L2 script is displayed while the corresponding L2 utterance is played acoustically. By comparing L1_{S1} and L1_{SS} using dynamic time warping (DTW), this method effectively captures sequential disfluency patterns that indicate listening breakdowns. Prior research has demonstrated that phonetic posteriorgram (PPG)-based DTW alignment between L1_{S1} and L1_{SS} can effectively yield unintelligibility annotations at the word, syllable, and phoneme levels [ 8 , 9 ] . Fig 1(b) illustrates the process for obtaining frame-wise disfluency labels on L2_{R} using the two-stage reverse shadowing approach. First, DTW is applied between the first shadowing speech L1_{S1} and the script-shadowing speech L1_{SS} to capture listening breakdowns in L1_{S1} . Then, by thresholding the edit distance between L1_{S1} and L1_{SS} , a disfluency label ( \mathbf{D} -Label) can be derived for both L1_{SS} and L1_{S1} . Finally, another DTW alignment between L1_{SS} and L2_{R} is used to map the \mathbf{D} -Label onto L2_{R} , thereby identifying the unintelligible frames. 2.2 Seq2seq voice conversion with self supervised speech representations Conventional voice conversion (VC) seeks to modify non-/para-linguistic features while preserving the linguistic content of the input speech. With the advent of end-to-end architectures, recent seq2seq VC models have achieved robust mappings between sophisticated source and target features. For example, [ 10 ] introduced a transformer-based VC model capable of sequentially mapping speech with variable durations. Meanwhile, limitations of autoregressive models, such as inaccurate duration prediction and repetitive artifacts, was addressed by proposing a non-autoregressive model using a conformer structure in [ 11 ] . More recent approaches combine Monotonic Alignment Search (MAS) with joint vocoder training [ 12 , 13 ] , which enhances the precision of duration modeling and improves prosody synthesis by reducing alignment errors. Advances in self-supervised learning have further enhanced VC by improving embedding features. Self-supervised speech representations (S3Rs) such as Wav2vec 2.0 [ 14 ] , HuBERT [ 15 ] , and WavLM [ 16 ] have set new benchmarks across diverse speech tasks [ 17 ] . Consequently, self-supervised speech representation-based VC (S3R-VC) has demonstrated notable gains in both linguistic accuracy and acoustic quality, while also proving effective for processing unintelligible speech such as electrolaryngeal speech enhancement [ 18 , 19 ] . Moreover, the combination of improved architectures and advanced embeddings has enabled novel mappings between source and target speech—such as shadowing-like speech generation [ 20 ] —even when the linguistic content of the two