Evaluating ASR Confidence Scores for Automated Error Detection in User-Assisted Correction Interfaces Conference: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan Extended Abstracts of the CHI Conference on Human Factors in Computi
Korbinian Kuhn, Verena Kersken, Gottfried Zimmermann · 2025
Read Evaluating ASR Confidence Scores for Automated Error Detection in User-Assisted Correction Interfaces Conference: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan Extended Abstracts of the CHI Conference on Human Factors in Computi by Korbinian Kuhn, Verena Kersken, Gottfried Zimmermann (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15124.txt.
Introduction
Korbinian Kuhn, Verena Kersken, and Gottfried Zimmermann test whether speech software can flag its own errors in a 2025 paper. Speech-to-text tools give each word a confidence score for how sure the tool is. In theory, low scores could point users to likely errors. The team checked that idea with model tests and a user study.
They tested end-to-end speech models and ran a study with 36 users. Scores did track accuracy to a degree, but error spotting was weak. Tools built on scores missed many errors or flagged many right words. Users did not fix text faster with the flags and did not find them helpful. The authors say better methods are needed to guide users.
Catalog record
| CREATOR | Korbinian Kuhn, Verena Kersken, Gottfried Zimmermann |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15124
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating ASR Confidence Scores for Automated Error Detection in User-Assisted Correction Interfaces Conference: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems; April 26-May 1, 2025; Yokohama, Japan Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’25), April 26-May 1, 2025, Yokohama, Japan DOI: 10.1145/3706599.3720038 ISBN: 979-8-4007-1395-8/2025/04 CCS: Human-centered computing Natural language interfaces CCS: Human-centered computing Empirical studies in HCI CCS: Human-centered computing Empirical studies in interaction design CCS: Human-centered computing Empirical studies in visualization CCS: Human-centered computing Empirical studies in accessibility CCS: Computing methodologies Speech recognition Korbinian Kuhn, Verena Kersken, Gottfried Zimmermann Abstract . Despite advances in Automatic Speech Recognition (ASR), transcription errors persist and require manual correction. Confidence scores, which indicate the certainty of ASR results, could assist users in identifying and correcting errors. This study evaluates the reliability of confidence scores for error detection through a comprehensive analysis of end-to-end ASR models and a user study with 36 participants. The results show that while confidence scores correlate with transcription accuracy, their error detection performance is limited. Classifiers frequently miss errors or generate many false positives, undermining their practical utility. Confidence-based error detection neither improved correction efficiency nor was perceived as helpful by participants. These findings highlight the limitations of confidence scores and the need for more sophisticated approaches to improve user interaction and explainability of ASR results. 1. Introduction Confidence scores are an additional output of an AI ( AI ) system and represent the degree of certainty a model has in its prediction. In ASR ( ASR ), these scores reflect the probability with which a spoken word is recognised correctly. Despite recent improvements in the accuracy of ASR ( Baevski et al. 2020 ; Radford et al. 2023 ) , transcripts often require manual editing to be intelligible ( Butler 2019 ; McDonnell et al. 2021 ; Seita et al. 2022 ) . Automatically highlighting transcription errors could help users to correct them more quickly and accurately. One approach is to use confidence scores to highlight low-confidence words to indicate potential recognition errors ( Luz et al. 2008b ; Luz et al. 2008a ; Berke 2017 ) . Such automated error detection could help users to correct errors more quickly and accurately. ASR models can provide word-level confidence scores, which are usually represented as numerical values between 0 and 1, with higher scores indicating greater confidence in the correctness of the result. To automatically classify words as correct or incorrect based on the confidence score, a threshold must be set that defines the minimum confidence level at which a word should be considered correct (e.g. all words above a score of 0.5). As this classification does not necessarily align with the actual correctness of a word, it can lead to false positives and missed detections. Ideally, the threshold is well balanced, assisting users to find major errors without distracting them with non- or minor errors. Confidence scores are mainly used for downstream ML ( ML ) tasks, but have also been proposed to assist users in correcting transcriptions ( Luz et al. 2008b ; Luz et al. 2008a ) and DHH ( DHH ) individuals in reading automated captions ( Berke 2017 ; Kafle et al. 2019 ) . However, research on their reliability and effectiveness for these tasks remains limited. Evidence from early studies suggests that they are either unreliable ( Suhm et al. 2001 ; Feng and Sears 2004 ; Vertanen and Kristensson 2008 ) or have a limited positive effect on user performance ( Burke et al. 2006 ) . As the architecture of ASR has changed fundamentally in recent years, findings from earlier evaluations may not be directly applicable to today’s systems. Transformers and RNN using attention mechanisms have largely replaced HMM ( HMM )-based models in modern ASR ( Chorowski et al. 2015 ; Chan et al. 2016 ; Kim et al. 2017 ; Gulati et al. 2020 ; Peng et al. 2022 ) , leveraging large-scale deep learning techniques and E2E ( E2E ) training ( Rasmus et al. 2015 ; Baevski et al. 2020 ; Yu et al. 2010a ) . These E2E models show increased robustness and accuracy ( Radford et al. 2023 ) but use differing confidence estimation techniques ( Park et al. 2020 ) and exhibit overconfidence ( Guo et al. 2017 ) . Although the use of confidence scores in E2E models for ML is an active area of research ( DeVries and Taylor 2018 ; Woodward et al. 2020 ; Kumar et al. 2020 ; Li et al. 2021 ; Tu et al. 2022 ) , their application in user interfaces may differ and has not yet been evaluated. This study explores whether confidence scores can reliably detect transcription errors and assist users in correcting transcripts. We comprehensively analyse current ASR models employing E2E architectures. We complement the quantitative evaluation with a user study applying the confidence-based error classification for manual correction. Our results show: • Classifiers using confidence scores achieve only moderate performance and cannot reliably detect transcription errors. • Confidence-based error detection neither improved correction efficiency nor proved useful to participants. 2. Background To contextualise previous HCI ( HCI ) research on confidence scores, it is important to understand the development of ASR models and corresponding confidence measures. This section first looks at the technical fundamentals of confidence scores and then at their applied use in HCI studies. 2.1. Confidence Measures in Machine Learning Confidence measures are used for various ASR downstream tasks, particularly ML methods like unsupervised ( Yu et al. 2010a ) , semi-supervised ( Chan and Woodland 2004 ; Huang et al. 2013 ; Park et al. 2020 ) , and active learning ( Yu et al. 2010b ; Huang et al. 2016 ; Li and Sethi 2006 ) . They are also used for speaker adaptation ( Del-Agua et al. 2018 ; Uebel and Woodland 2001 ) , keyword spotting ( Benayed et al. 2003 ; Keshet et al. 2009 ; Seigel et al. 2013 ) , and dialogue systems ( Hazen et al. 2002 ; Tur et al. 2005 ) . In HMM -based ASR , confidence is often predicted using binary classifiers on decoding features such as word posterior probabilities ( Evermann and Woodland 2000 ) , word trellis stability ( Sanchis et al. 2003 ) , normalised likelihoods, and language model score ( Willett et al. 1998 ) . These features are classified using simple linear mappings, such as decision trees ( Evermann and Woodland 2000 ) and Gaussian mixture classifiers ( Chigier 1992 ) , or sequence models such as conditional random fields ( Seigel and Woodland 2011 ) , neural networks ( Weintraub et al. 1997 ; Schaaf and Kemp 1997 ) , and recurrent neural networks ( Kalgaonkar et al. 2015 ; Li et al. 2019 ; Kastanos et al. 2020 ) . ASR systems have evolved from HMM and hybrid designs to E2E models, which rely on an encoder-decoder architecture ( Chan et al. 2016 ; Kim et al. 2017 ; Dong et al. 2018 ) . A common confidence measure for these models, the softmax probability ( Park et al.