Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition

Peng, Yanqiao, Zixuan, Qinyuan, Xingjian, Xipeng, Wupeng, Zhifu, Xiangang, Kai, Xie · 2026

Read Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition by Peng, Yanqiao, Zixuan, Qinyuan, Xingjian, Xipeng, Wupeng, Zhifu, Xiangang, Kai, Xie (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09121.txt.

Introduction

Wang and ten colleagues build an agentic framework for interactive speech recognition. Word error rate counts every word equally and often misses whether a sentence means the right thing. Interactive correction, a normal part of human conversation, has rarely been studied systematically for speech recognition. The framework uses a language model as judge for semantic-aware scoring beyond token accuracy. A language-model-driven agent simulates multi-turn interaction, refining outputs through semantic feedback. Tests on English, Chinese, and code-switching benchmarks show gains in fidelity and correction ability.

Catalog record

CREATORPeng, Yanqiao, Zixuan, Qinyuan, Xingjian, Xipeng, Wupeng, Zhifu, Xiangang, Kai, Xie
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21730
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.09121

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition Thanks: Project page: https://interactiveasr.github.io/ . Live demo: https://i-asr.sjtuxlance.com/ Peng, Yanqiao, Zixuan, Qinyuan, Xingjian, Xipeng, Wupeng, Zhifu, Xiangang, Kai, Xie Abstract Recent years have witnessed remarkable progress in automatic speech recognition (ASR), driven by advances in model architectures and large-scale training data. However, two important aspects remain underexplored. First, Word Error Rate (WER), the dominant evaluation metric for decades, treats all words equally and often fails to reflect the semantic correctness of an utterance at the sentence level. Second, interactive correction—an essential component of human communication—has rarely been systematically studied in ASR research. In this paper, we integrate these two perspectives under an agentic framework for interactive ASR. We propose leveraging LLM-as-a-Judge as a semantic-aware evaluation metric to assess recognition quality beyond token-level accuracy. Furthermore, we design an LLM-driven agent framework to simulate human-like multi-turn interaction, enabling iterative refinement of recognition outputs through semantic feedback. Extensive experiments are conducted on standard benchmarks, including GigaSpeech (English), WenetSpeech (Chinese), the ASRU 2019 code-switching test set. Both objective and subjective evaluations demonstrate the effectiveness of the proposed framework in improving semantic fidelity and interactive correction capability. We will release the code to facilitate future research in interactive and agentic ASR. Wang Zhu † Jiang Chen Zhao Qiu Wang Gao Li Yu Chen keywords speech recognition, human-computer interaction, LLM agent † † address: 1 X-LANCE Lab, Shanghai Jiao Tong University 2 The Chinese University of Hong Kong, Shenzhen 3 Xi’an Jiaotong University 4 Fudan University 5 Tongyi Fun Team, Alibaba Group † † email: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] † † footnotetext: This work was conducted during an internship at Tongyi Fun Team, Alibaba Group. Figure 1: Traditional vs. Interactive ASR paradigms. (Top) Traditional systems struggle with named entities (e.g., ”Night” vs. ”Knight”); (Bottom), our proposed Interactive ASR can take user’s spoken language instructions (e.g., ”starts with a K”) as the feedback to update and correct ASR results. 1 Introduction Automatic speech recognition (ASR) plays a pivotal role in human–computer interaction by enabling computers to understand users’ intent through speech. In recent years, ASR technologies have achieved remarkable progress, driven by advances in both model architectures and large-scale training data. Extensive research has explored a variety of modeling paradigms, ranging from end-to-end approaches [ 8 , 9 , 3 , 15 ] , to more recent large language model (LLM)-based frameworks [ 24 , 22 , 21 , 5 , 7 , 12 ] . Meanwhile, scaling laws in both model capacity and training data have proven highly effective in further advancing ASR performance [ 2 , 25 , 1 , 18 ] . However, despite the rapid progress in ASR, several important aspects remain underexplored. The first concerns the evaluation metrics used for ASR systems. For decades, Word Error Rate (WER) has been widely adopted to measure the discrepancy between recognition hypotheses and reference transcriptions by aligning words in the two sequences. Despite its simplicity and objectivity, this de facto metric has several well-known limitations. In particular, WER treats all words equally and assigns the same penalty to each recognition error, while different types of words may affect sentence semantics differently [ 16 ] . For example, errors in function words often have limited impact, whereas misrecognition of critical content words such as named entities can significantly alter the intended meaning. As a result, WER may fail to adequately reflect the semantic impact of recognition errors. Meanwhile, large language models (LLMs) have demonstrated strong semantic understanding capabilities and can often infer or correct minor recognition errors during interaction [ 11 ] . This suggests that future evaluation of ASR systems should move beyond word-level accuracy and focus more on errors that may mislead downstream language understanding. The second issue lies in the integration of user feedback. In human communication, clarification and correction through interaction are common when key information is ambiguous or misunderstood. However, such interactive correction has been largely overlooked in human–computer interaction. Most current ASR systems cannot revise their recognition outputs once errors occur, even when users explicitly point them out, which may significantly degrade user experience, particularly in on-screen scenarios. Moreover, the inherent ambiguity of spoken language—such as homonyms in names or recognition errors caused by accents or background noise—further motivates the development of ASR systems that can interact with users to clarify and correct recognition results. To address these challenges and adapt ASR systems to the LLM era, we propose Interactive ASR , a framework that integrates semantic-aware evaluation and user interaction. First, we revisit LLM-as-a-Judge [ 30 ] for ASR evaluation and demonstrate that modern LLMs can achieve strong agreement with human judgments when assessing the semantic consistency between ASR outputs and reference transcriptions. This motivates our proposed Sentence-level Semantic Error Rate ( S^{2} ER) as a complementary metric to WER to evaluate semantic coherence. In addition, we design an LLM-driven agentic framework that enables ASR systems to interact with users and leverage feedback for iteratively correcting recognition errors. Our key contributions are summarized as follows: • New ASR Semantic Metric ( S^{2}ER ): We propose S^{2}ER , a novel evaluation metric that leverages LLMs as judges to assess ASR semantic success. Through human evaluation, we demonstrate a strong correlation between LLM judgments and human preferences in terms of semantic consistency. • Agentic Correction Framework: We develop an LLM-simulated interactive correction framework to iteratively improve ASR performance. • Robust Generalization and Validation: Extensive experimental validation across diverse linguistic standard benchmarks, demonstrates the framework’s broad applicability. 2 Related Works While WER has long served as the standard metric for ASR evaluation, its inherent design of assigning equal weight to all words fails to capture critical semantic errors. To address this, several semantic-aware metrics have been proposed. Semantic WER [ 16 ] introduced dynamic weighting, which utilizes Named Entity Recognition to extract keywords, assigning higher weights to critical entities and lower weights to filler words during error calculation. Moving to embedding-based evaluation, SemDist [ 10 ] utilized RoBERTa-based sentence embeddings to measure semantic similarity beyond literal overlap. Most recently, LASER [ 13 ] leveraged LLMs to assign graded penalties based on error severity (e.g., ignoring colloquial variations while penalizing meaning changes). Unlike these continuous scoring metrics, our LLM-as-a-Judge adopts a binary functional criterion, acting as a strict gatekeeper to determine if the user’s intent is executable. Regarding error correction based on human feedback, traditional ASR systems have explored several approaches, but they often rely on rigid interactions. Early methods utilized Multimodal Interfaces [ 20 ] , requiring users to manually select alternatives from an N-best list via a GUI or keyboard, which disrupts the hands-free nature of voice interactions. Another approach is Acoustic Respeaking [ 19 ] , where users simply repeat the misrecognized utterance, which is highly inefficient. In contrast, the NLP community has successfully leveraged natural language feedback for error resolution. For instance, agentic frameworks like ReAct [ 27 ] enable LLMs to iteratively reason and refine their outputs based on human or environmental feedback. Inspired by these advancements, our work introduces this interactive paradigm to ASR. Instead of relying on manual edits or rigid repetition, our framework allows users

Read the complete text