Evaluating Speech-to-Text Systems with PennSound
Jonathan, Mark, Neville, James · 2025
Read Evaluating Speech-to-Text Systems with PennSound by Jonathan, Mark, Neville, James (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05702.txt.
Introduction
Jonathan Wright and three co-authors grade speech-to-text systems on poetry audio in a 2025 paper. The sound bank PennSound holds the world's largest set of poetry reads and talks, with wild swings in tape grade and voice style. Such range makes it a fair test field for text-from-speech tools.
Near ten hours were drawn at random and typed by trained hands as the truth set. Eight trade and open tools were scored on word slips and speaker splits. The top trade tool edged the field, and the top open one kept close when made-up text was dodged. Gaps were slim all round.
Catalog record
| CREATOR | Jonathan, Mark, Neville, James |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 15 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-05702
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Speech-to-Text Systems with PennSound Jonathan, Mark, Neville, James Abstract A random sample of nearly 10 hours of speech from PennSound, the world’s largest online collection of poetry readings and discussions, was used as a benchmark to evaluate several commercial and open-source speech-to-text systems. PennSound’s wide variation in recording conditions and speech styles makes it a good representative for many other untranscribed audio collections. Reference transcripts were created by trained annotators, and system transcripts were produced from AWS, Azure, Google, IBM, NeMo, Rev.ai, Whisper, and Whisper.cpp. Based on word error rate, Rev.ai was the top performer, and Whisper was the top open source performer (as long as hallucinations were avoided). AWS had the best diarization error rates among three systems. However, WER and DER differences were slim, and various tradeoffs may motivate choosing different systems for different end users. We also examine the issue of hallucinations in Whisper. Users of Whisper should be cautioned to be aware of runtime options, and whether the speed vs accuracy trade off is acceptable. Wright Liberman Ryant Fiumara University of PennsylvaniaUSA keywords speech recognition, benchmarks, speech corpora † † email: [email protected], [email protected], [email protected], [email protected] 1 Introduction PennSound 1 1 1 https://writing.upenn.edu/pennsound/ is the world’s largest online collection of poetry readings and discussions 2 2 2 Charles Bernstein, co-creator, https://www.youtube.com/watch?v=mClvCzCOHaE . The recordings are free to download for non-commercial purposes. This work is part of an effort to evaluate the efficacy of current speech-to-text technology in indexing this collection. The varied nature of the PennSound collection, which includes interviews and discussions as well as readings, means that this evaluation may also be relevant to other large-scale audio datasets. We chose a random sample of 100 five-minute or longer selections 3 3 3 See supplementary materials for details. from this data to evaluate several leading speech-to-text systems. The sample includes a broad range of conditions, including varying recording quality, number of speakers, and speaking style. 2 Methods The PennSound archive includes over 30,000 links to mp3s, although many of these files are individual poems excerpted from larger recordings. Over two thousand files downloaded from the site averaged 37.9 minutes in duration, for a total of 1,295 hours, out of a total of more than 6,000 hours. The histogram in Figure 1 shows the distribution of durations for this subset of the collection. Figure 1: Distribution of full length recording durations. A random selection of 100 sources was made from 1,869 .mp3 URLs on the PennSound website that fall under the Authors page, and have the word Complete in the file name, indicating a complete recording. Two files outside this list, from Close Listening and Poem Talks , were arbitrarily added to ensure examples with multiple speakers. Speech activity labels were produced using an HMM-based speech activity detector (SAD) trained on English interviews ( ldc-bpcsad [ 1 ] ) . Starting from the beginning of a speech segment near the midpoint of each full recording, a stretch of audio was extracted such that at least 5 minutes of speech was included, based on the SAD labels. These 100 clips became the data for the experiments described below. The sample includes introductions, readings, question-and-answer sessions, and discussions. Reference transcripts were created by a team of trained annotators using a set of transcription guidelines 4 4 4 See supplementary materials. and an online transcription platform [ 2 ] . Several speech-to-text systems were used to generate hypothesis transcripts, including five commercial cloud services: Amazon Transcribe 5 5 5 https://aws.amazon.com/transcribe/ (aws), Microsoft Azure AI Speech 6 6 6 https://azure.microsoft.com/en-us/products/ai-services/ai-speech/ (azure), Google Cloud Speech-to-Text 7 7 7 https://cloud.google.com/speech-to-text?hl=en (google), IBM Watson Speech to Text 8 8 8 https://www.ibm.com/products/speech-to-text (ibm), and Rev.ai 9 9 9 https://www.rev.ai/async (rev). NVIDIA’s NeMo 10 10 10 https://github.com/NVIDIA/NeMo (nemo), Open AI’s Whisper 11 11 11 https://github.com/openai/whisper (whisper), and the related Whisper CPP 12 12 12 https://github.com/ggerganov/whisper.cpp (whispercpp), three open source systems, were also run locally. The mp3s were converted to \SI 16 \kilo \hertz mono using SoX 13 13 13 https://sourceforge.net/projects/sox/ , using the --norm=-1 option to normalize the levels. Files were then fed to the systems as wav or flac depending on requirements. Parameter values used with the systems are provided in the supplementary material. The corresponding parameters are described in the documentation for aws 14 14 14 https://docs.aws.amazon.com/transcribe/latest/APIReference/API_StartTranscriptionJob.html , azure 15 15 15 https://learn.microsoft.com/en-us/dotnet/api/microsoft.cognitiveservices.speech.speechconfig?view=azure-dotnet , google 16 16 16 https://cloud.google.com/speech-to-text/v2/docs/transcription-model , ibm 17 17 17 https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-use , nemo 18 18 18 https://docs.nvidia.com/nemo-framework/user-guide/24.09/nemotoolkit/asr/intro.html , rev 19 19 19 https://docs.rev.ai/api/asynchronous/reference/#operation/SubmitTranscriptionJob , whisper 20 20 20 https://github.com/openai/whisper , whispercpp 21 21 21 https://github.com/ggerganov/whisper.cpp . Following previous work [ 3 , 4 , 5 , 6 ] , Word Error Rate (WER) is used to evaluate word recognition by the systems. WERs were calculated with NIST’s Scoring Toolkit (SCTK) 22 22 22 https://github.com/usnistgov/SCTK . The sctk command calculates the edit distance between reference transcripts and hypothesis transcripts, and the normalized edit distance is the WER. We also use hubscr.pl from SCTK, which calls sctk after normalizing spelling. Scoring in this way also involves comparing timestamps, which are not of primary interest here (see Future Work), so the reference transcripts, which have segment boundaries, are converted into single segment transcripts. As three cloud services 23 23 23 The chosen settings for google and ibm don’t support diarization. also produce speaker labels, we additionally report diarization error rate (DER) [ 7 ] , calculated using the dscore tool [ 8 ] used by the DIHARD challenges [ 9 ] . 3 Results 3.1 Word Recognition Word error rates from SCTK for the full sample are shown in Table 1 . Table 1: Word Error Rates System WER aws 9.8 azure 10.2 google 11.0 ibm 14.4 nemo 10.9 rev 9.0 whisper 9.5 whispercpp 10.9 Other work suggests these WERs are better than what is necessary for effective information retrieval [ 10 ] . As mentioned earlier, the nature of the files varies quite a bit, from clear readings, to difficult to understand discussions, and while the sample is random, the distribution of these variables is not balanced. The total WER may obscure performance details, and individual WERs per file provide detail on how the systems vary across files. Figure 2 plots all the individual WERs; each system’s WERs have been sorted separately. The complete table of WERs by sample and system is available in the Supplementary Materials. Figure 2: WERs, Sorted Separately 3.2 Diarization Three systems, aws, azure, and rev, provide speaker labels, which we can evaluate with the dscore package; DER scores are shown in Table 2 . Table 2: DERs System DER aws 14.52 azure 15.40 rev 15.49 As with WERs, we can look at performance by recording. Figure 3 plots the DERs for individual recordings, with each system sorted separately. Figure 3: DERs, Sorted Separately 4 Discussion Although ibm lags slightly, the performance of the systems based on WER alone is effectively the same for end users. There is a slight tendency for Whisper to do best on the easier recordings, and a slight tendency for Rev.ai to do best on the harder recordings. AWS, Azure, and Rev.ai are effectively the same based on DER as well. Whisper is an attractive system, but only if hallucinations are avoided. Whisper