Evaluation of Automatic Speech Recognition Using Generative Large Language Models
Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek, Shiran Liu, Mickael Rouvier, Jane Wottawa, Richard Dufour · 2026
Read Evaluation of Automatic Speech Recognition Using Generative Large Language Models by Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek, Shiran Liu, Mickael Rouvier, Jane Wottawa, Richard Dufour (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21928.txt.
Introduction
Bañeras-Roux and eight colleagues evaluate speech recognition with generative large language models, beyond Word Error Rate, a metric blind to meaning. They test three approaches: picking the best hypothesis of two, computing semantic distance with generative embeddings, and classifying errors by type. On the HATS dataset, the best models reach 92 to 94 percent agreement with human judges on choice, versus 63 percent for Word Error Rate. Decoder-based vectors perform as well as encoder models. The authors see promise for clear, semantic speech checks.
Catalog record
| CREATOR | Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek, Shiran Liu, Mickael Rouvier, Jane Wottawa, Richard Dufour |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22884 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.21928
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluation of Automatic Speech Recognition Using Generative Large Language Models Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek, Shiran Liu, Mickael Rouvier, Jane Wottawa, Richard Dufour Abstract Automatic Speech Recognition (ASR) is traditionally evaluated using Word Error Rate (WER), a metric that is insensitive to meaning. Embedding-based semantic metrics are better correlated with human perception, but decoder-based Large Language Models (LLMs) remain underexplored for this task. This paper evaluates their relevance through three approaches: (1) selecting the best hypothesis between two candidates, (2) computing semantic distance using generative embeddings, and (3) qualitative classification of errors. On the HATS dataset, the best LLMs achieve 92–94% agreement with human annotators for hypothesis selection, compared to 63% for WER, also outperforming semantic metrics. Embeddings from decoder-based LLMs show performance comparable to encoder models. Finally, LLMs offer a promising direction for interpretable and semantic ASR evaluation. 1 Introduction Automatic Speech Recognition (ASR) is a fundamental technology for voice control applications, transcription, and accessibility. Reliable evaluation of ASR systems is essential for measuring progress and guiding model improvement. Historically, Word Error Rate (WER) has dominated this landscape, but this metric has well-known limitations: it is rigid and case-sensitive, not necessarily capturing information important for the task or for human perception. Facing these challenges, several alternatives have been proposed. Semantic metrics 13 ; 28 based on word embeddings have shown promising performance, particularly those exploiting contextual encoder models like BERT 7 . These approaches better capture the semantic nuances of language, explaining their better correlation with human judgments. However, while encoders have been frequently explored for this task 2 ; 13 ; 16 ; 9 , decoder-based Large Language Models (LLMs) – such as GPT 21 , Llama 26 , or Gemma 24 – have received little attention. Yet these models have demonstrated remarkable and versatile capabilities across a wide range of tasks 1 ; 15 ; 20 , suggesting they could offer innovative perspectives for ASR evaluation. This study explores precisely the capabilities of decoder-based LLMs in the specific context of evaluating automatic speech recognition systems, along two complementary axes. The first axis focuses on the “LLM as judge” paradigm: can we leverage the deep contextual and semantic understanding of LLMs to directly evaluate the quality of transcription hypotheses? The second axis examines the quality of vector representations extracted from these models: how do embeddings from decoder LLMs compare to those from encoders in capturing relevant semantic similarities? Our experiments rely on the HATS dataset 5 , which provides human annotations of speech recognition errors and has shown that semantic metrics correlate significantly better with human perception than WER. We evaluate (1) the capacity of LLMs to select the best hypothesis among two candidates, (2) the effectiveness of different pooling strategies applied to LLM embeddings to build a semantic similarity metric, and (3) the capacity of LLMs to assign qualitative labels to transcription errors. Our results reveal that the best LLMs not only outperform WER and CER, but also sophisticated semantic metrics based on encoders, while offering better interpretability of errors. 2 Related Work 2.1 Limitations of Word Error Rate As mentioned previously, Word Error Rate has been the reference metric for evaluating automatic speech recognition systems. Yet WER presents significant limitations well-documented in the literature 27 ; 18 ; 10 . First, this metric operates at the lexical level without any consideration of semantic context, treating errors that severely affect comprehension identically to minor errors with no impact on meaning. Second, WER does not necessarily correlate with downstream tasks 8 that depend on the produced transcriptions, suggesting that WER improvement does not guarantee perceptible improvement for the end user. Finally, several studies have demonstrated poor correlation between WER and human perception of transcription quality 14 ; 25 , questioning its usefulness for evaluating the real performance of ASR systems intended for human reading. These limitations have important practical implications. Indeed, studies have shown that the ranking of ASR systems can differ significantly depending on the metric used for evaluation 3 . For a specific objective – for example, generating subtitles to be read by humans – it may be more relevant to use a metric that correlates better with human perception rather than optimizing a WER that does not capture relevant perceptual nuances 12 ; 19 . 2.2 Semantic Metrics Based on Embeddings Facing WER’s limitations, several alternatives have been proposed. Among these, semantic metrics based on word embeddings have proven particularly promising. These approaches exploit vector representations of language obtained by word embedding or context models, allowing capture of semantic similarities between reference and transcription hypothesis. A first generation of these metrics exploited static word embeddings to improve ASR evaluation in machine translation context 16 . More recently, the emergence of powerful contextual models, particularly bidirectional encoders like BERT and its multilingual variants, has enabled development of more sophisticated semantic metrics. These contextual models offer richer representations, sensitive to word position and the context surrounding each linguistic unit. Among notable semantic approaches, the SemDist metric 13 proposes a distance derived from cosine similarity between embeddings of a reference and a hypothesis. This metric has demonstrated significantly better correlation with human perception compared to WER, validating the hypothesis that semantics is a key factor for perceptual evaluation of transcription quality. 17 defines the ideal metric for automatic speech recognition on several criteria including interpretability. The problem is that semantic metrics are primarily constructed from cosine similarities between embeddings, producing scores such as: 0.62, 0.75, or 0.81. Unlike error percentages (as WER and CER do), these scores are poorly interpretable. To address this issue, some work 9 ; 22 has proposed a semantic metric that can be adapted to a downstream task and could be interpreted as a weighted word error rate based on the semantic and lexical severity of an error. Other work 4 has also proposed paradigms for interpreting ASR errors based on their severity and impact on semantic metrics using contextual embeddings. These approaches offered more granular understanding of errors, allowing identification of which error types most affect perceived transcription quality. 2.3 Human Perception and Interpretable Evaluation A complementary research axis has focused on explicit alignment between automatic metrics and human perception. Several works have proposed evaluation models based on human perception 11 ; 23 . Empirical studies have collected human annotations to evaluate how speakers perceive the quality of automatic transcriptions. These annotations enabled creation of datasets like HATS, a French dataset that integrate explicit human judgments on error quality, allowing validation and comparison of different evaluation metrics. This study showed greater correlation with human perception than WER, confirming another study 14 conducted on private datasets integrating human perception. Another recent study 25 also demonstrated using human annotations that character error rate is more relevant than WER in a multilingual setting. 3 Experiments We rely on results reported in the HATS dataset to establish a reference point for metric evaluation. The table 1 presents the percentage of agreement between different automatic metrics and human judgments. To better interpret these results, three subsets of the corpus are considered, constructed based on the level of agreement between human annotators: (i) examples with strict 100% agreement, (ii) those with at least 70%