Swedish Whispers; Leveraging a Massive Speech Corpus for Swedish Speech Recognition
Leonora, Faton, Robin, Justyna, Agnes · 2025
Read Swedish Whispers; Leveraging a Massive Speech Corpus for Swedish Speech Recognition by Leonora, Faton, Robin, Justyna, Agnes (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17538.txt.
Introduction
Leonora Vesterbacka and four co-authors improve Swedish speech recognition in a 2025 paper. Swedish is a mid-sized language, so global speech models train on little of it. That gap leaves room for large gains from focused tuning on Swedish data.
The team fine-tuned a family of Whisper models on a Swedish speech collection of record size and variety. Scores improved across all model sizes over the base system. Their best model cut the word error rate by nearly half on average. Tests covered three public data sets for Swedish speech. The work shows how modest languages can gain strong recognition tools without building models from scratch.
Catalog record
| CREATOR | Leonora, Faton, Robin, Justyna, Agnes |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17538
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Swedish Whispers; Leveraging a Massive Speech Corpus for Swedish Speech Recognition Leonora, Faton, Robin, Justyna, Agnes Abstract This work presents a suite of fine-tuned Whisper models for Swedish, trained on a dataset of unprecedented size and variability for this mid-resourced language. As languages of smaller sizes are often underrepresented in multilingual training datasets, substantial improvements in performance can be achieved by fine-tuning existing multilingual models, as shown in this work. This work reports an overall improvement across model sizes compared to OpenAI’s Whisper evaluated on Swedish. Most notably, we report an average 47% reduction in WER comparing our best performing model to OpenAI’s whisper-large-v3 , in evaluations across FLEURS, Common Voice, and NST. Vesterbacka Rekathati Kurtz Sikora Toftgård National Library of SwedenSweden keywords Automatic speech recognition, Speech-to-text, Whisper † † email: {leonora.vesterbackaolsson,faton.rekathati,robin.kurtz,justyna.sikora,agnes.toftgard} @kb.se 1 Introduction The abundance of openly available audio data in English enables pre-training and fine-tuning of speech-to-text models on hundreds of thousands to millions of hours of recorded speech. As a result, automatic speech recognition (ASR) systems are approaching human level robustness in English. Other languages’ performance in multilingual speech recognition tends to stand in proportion to the amount of data included from the language – or the language family – in question. For low- to mid-resource languages with fewer speakers, such as Swedish, the amount of openly available data may be limited, and as a consequence these languages tend to be underrepresented in large-scale efforts to train multilingual speech recognition systems. This results in suboptimal performance of ASR applications, such as voice-based customer service, and automatic captioning. Language models and ASR models are often trained on massively multilingual datasets. By continuing training these multilingual models for a specific language, the performance is greatly improved compared to the performance of multilingual models. This is necessary for low- and mid-resource languages to remain competitive and sovereign in the rapid AI development of today. Similarly to previous work for Norwegian [ 1 ] , we fine-tune Whisper for all canonical sizes on our massive dataset for Swedish. Large collections of audio recordings maintained by cultural heritage institutions, such as the National Library of Sweden, and government agencies can potentially bridge some of the existing speech recognition performance gaps between Swedish and higher resourced languages. By scaling up the amount of training data, one can noticeably improve Swedish speech recognition. In this work, KBLab [ 2 ] at the National Library of Sweden have a have constructed a massive speech corpus with an emphasis on all variations of spoken Swedish, which is used to train speech recognition models for Swedish. 2 Automatic speech recognition Automatic Speech Recognition models translate speech (i.e. spoken language) into text (i.e. written language). The form of the output may vary between phonetic transcriptions, raw text without punctuation, and formatted sentences with punctuation. 2.1 Model architecture Whisper [ 3 ] is a multilingual ASR model with additional translation capabilities. By relaxing the quality standard of the labeled training data, vast amounts of training data are unlocked. The authors show that even with imperfectly labeled training data, such as subtitles, it is still possible to achieve speech recognition performance approaching human-level robustness in English. Using the latest official Whisper models as a starting point, we fine-tune them on our herewith presented corpus for Swedish speech. The Whisper model architecture is an encoder-decoder transformer [ 4 ] . The input audio is split up into \SI 30 \second chunks, resampled to \SI 16 \kilo \hertz , and transformed to an 80-channel log-magnitude Mel spectrogram (128-channel for large-v3 ). Feature normalization is applied along with a two-layered 1-dimensional convolutional encoder with a GELU activation function, followed by sinusoidal position embeddings, and transformer encoder blocks. We employ a Byte Pair Encoding (BPE) dropout [ 5 ] of 0.2, which was shown to have beneficial regularizing effects for the large model size [ 1 ] . An activation dropout of 0.1 is also applied. The model is trained using the PyTorch implementation by Hugging Face that is detailed in the Whisper fine-tuning Event 1 1 1 https://github.com/huggingface/community-events/tree/main/whisper-fine-tuning-event . We adapt this script and add functionality to train with timestamps as well as with previous contexts as a prompt. 3 A massive Swedish speech corpus OpenAI’s whisper-large-v3 2 2 2 https://huggingface.co/openai/whisper-large-v3 has been trained on \SI 680000 \hour of multilingual speech, of which \SI 2119 \hour are in Swedish. While [ 3 ] report that all increases in dataset size lead to improved performance, the improvement slightly stagnates after a dataset size of around \SI 50000 \hour . With these numbers in mind, a dataset consisting of \SI 50000 \hour of transcribed Swedish was constructed. Not only is the amount of data important to training ASR models, but also language variation. One common drawback of using the multilingual Whisper is its inability to recognize variations in a lower resourced language, such as dialects. While dialects are generally more difficult to correctly transcribe, given their decreased occurrence in the training data, this is especially noticeable for low- and mid-resource languages. Models trained on material from the web, which can be assumed for models such as Whisper, have only learnt to recognize speakers that are commonly represented in web material, e.g. YouTube videos. Training on this type of material excludes many varieties of speech that exists in a population. By actively including data sources that represent other types of speech, we increase the probability of developing a model that performs better for a wider variety of dialects, sociolects, age groups, and accents spoken by non-native Swedish speakers. In this work, KBLab at the National Library of Sweden has collaborated with the Riksdag administration (i.e. the parliament administration in Sweden), Sweden’s Television (SVT), and the Institute for Language and Folklore (Institutet för språk och folkminnen), and together we have been able to construct a transcribed Swedish dataset of unprecedented size and variability. These dataset types can be split into four categories, which are presented below. 3.1 Subtitles Subtitles have traditionally existed for Swedish television to subtitle non-Swedish TV broadcasts. Swedish closed captions for Swedish spoken content have become increasingly available as a means to aid accessibility for the hearing impaired. The Swedish subtitling tradition tends to condense the content rather than provide verbatim transcriptions, in order to provide the viewer enough time to read. In our filtering, we select only Swedish TV broadcasts and web content where metadata indicates the subtitle track is in Swedish. To ensure the speech is also in Swedish, we perform language detection using OpenAI’s whisper-large-v3 . The final subtitle dataset is thus constructed out of audio identified as Swedish paired with Swedish subtitles. 3.1.1 SMDB With access to the Swedish Media Database (SMDB) at the National Library of Sweden, we extracted audio and subtitles from 15 Swedish TV channels, in a \SI 24 \hour stream format, without any program metadata, from 2021 to 2023, resulting in more than \SI 150000 \hour of raw broadcasts. Of the \SI 63000 \hour of audio matching the subtitles, around 80% were duplicates. From this deduplicated subset, 20% did not contain Swedish speech. Finally, another 30% of chunks were dropped, not reaching the desired quality for training. 3.1.2 SVT In contrast to the SMDB dataset,