What do self-supervised speech models know about Dutch? Analyzing advantages of language-specific pre-training
Marianne, Hosein, Charlotte, Gaofei, Willem, Martijn · 2025
Read What do self-supervised speech models know about Dutch? Analyzing advantages of language-specific pre-training by Marianne, Hosein, Charlotte, Gaofei, Willem, Martijn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00981.txt.
Introduction
Speech models that learn from raw audio pick up the sounds of the tongue they train on. A team led by Marianne de Heer Kloots asks how much Dutch such a model learns, in a 2025 study. They probe the inner layers of a self-taught speech model for Dutch sound and word traits.
Models trained only on Dutch held richer Dutch traits than ones trained on English or on many tongues at once. The edge showed up in trained probes and, in part, in zero-shot tests. That edge also matched scores on Dutch speech-to-text tasks. The results argue for language-specific training where good data exists.
Catalog record
| CREATOR | Marianne, Hosein, Charlotte, Gaofei, Willem, Martijn |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00981
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What do self-supervised speech models know about Dutch? Analyzing advantages of language-specific pre-training Marianne, Hosein, Charlotte, Gaofei, Willem, Martijn Abstract How language-specific are speech representations learned by self-supervised models? Existing work has shown that a range of linguistic features can be successfully decoded from end-to-end models trained only on speech recordings. However, it’s less clear to what extent pre-training on specific languages improves language-specific linguistic information. Here we test the encoding of Dutch phonetic and lexical information in internal representations of self-supervised Wav2Vec2 models. Pre-training exclusively on Dutch improves the representation of Dutch linguistic features as compared to pre-training on similar amounts of English or larger amounts of multilingual data. This language-specific advantage is well-detected by trained clustering or classification probes, and partially observable using zero-shot metrics. Furthermore, the language-specific benefit on linguistic feature encoding aligns with downstream performance on Automatic Speech Recognition. de Heer Kloots Mohebbi Pouw Shen Zuidema Bentum University of AmsterdamThe Netherlands Tilburg UniversityThe Netherlands Radboud UniversityNetherlands keywords interpretability, self-supervised learning, language-specificity, speech recognition † † email: [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] 1 Introduction In recent years, self-supervised learning (SSL) algorithms have been demonstrated to learn powerful representations of spoken language, in terms of both their downstream task performance and the richness of their embedding spaces. Despite being trained only on unlabeled speech recordings, neural SSL models vastly outperform acoustic baselines at encoding various levels of linguistic structure, including phonological [ 1 , 2 ] , syllabic [ 3 ] and lexical [ 4 ] features. Understanding the linguistic information these models encode is relevant not only for engineers interpreting model behaviour, but also for cognitive scientists studying language acquisition from spoken input [ 5 , 6 ] . Despite the wide interest for interpreting SSL model functioning, research has so far primarily focused on English linguistic features encoded by English-only or multilingually pre-trained models, with some notable exceptions [ 7 , 8 , 9 , 10 ] . As a result, it remains somewhat unclear whether measures detecting linguistic structures in SSL models rely on truly language-specific representations or more language-general acoustic correlates. For downstream automatic speech recognition (ASR), both multi- and monolingually trained Wav2Vec2 [ 11 ] models are known to generalize well even to languages not present in their training data [ 12 , 13 ] . This could indicate that what these models learn about speech is not typically language-specific [ 14 ] , in line with findings that internal activations of Wav2Vec2 models do not seem strongly influenced by the model’s training language in distinguishing native vs. non-native phoneme contrasts [ 10 ] . However, the same models do show distinct patterns in predicting brain activity for their pre-training language vs. a different language [ 15 ] , and crosslingual comparisons of Wav2Vec2 models trained on tonal vs. non-tonal languages have found language-specific advantages in the representation of suprasegmental features [ 7 , 8 ] . A key challenge in distinguishing language-specific vs. language-general speech processing by neural SSL models is the variability in measures and datasets used to identify language-specific representations. While some studies use ‘zero-shot’ embedding space distance metrics to measure the saliency of native vs. non-native phoneme contrasts [ 10 ] , other studies use trained classifiers to detect how well language-specific information can be decoded from internal representations [ 7 , 8 ] . Hence, the differences in reported results could be driven by differences in the methodologies used for testing the models’ linguistic capabilities. Figure 1: A set of high front vowels (phonemically distinctive in Dutch) is most discriminably represented in a model trained exclusively on Dutch. Data points are vowel occurrences sampled across 3 speakers in Multilingual LibriSpeech. Here, we aim to investigate language-specific representations of Dutch linguistic information in self-supervised speech models, as well as the effects of different analysis pipelines on the detection of such information. For this purpose, we curated the SSL-NL evaluation set of publicly available Dutch speech data and alignments 1 1 1 We release the SSL-NL dataset at http://doi.org/10.5281/zenodo.15548947 . We designed the SSL-NL set to test the encoding of Dutch phonetic and lexical features in SSL speech representations, while allowing for comparisons across different analysis methods. We also pre-train a Wav2Vec2 model on the Dutch language exclusively 2 2 2 We release Wav2Vec2-NL along with its training manifest and configuration at http://doi.org/10.5281/zenodo.15550628 as an additional resource for studying Dutch-specific speech representations. Comparing the Dutch model to a monolingual English and a multilingual model, we find that both phone- and word-level features are best encoded in the Dutch model (e.g. Figure 1 ), which is also reflected in its downstream performance on Automatic Speech Recognition. Notably, the detection of language-specific benefits varies across analysis measures and datasets, underscoring the critical role of data selection and analysis methods in assessing the impact of language-specific pre-training on self-supervised speech representations. 2 Models To investigate the effect of language-specific pre-training on linguistic feature encoding, we compare three Wav2Vec2 models with identical architectures (7 CNN + 12 Transformer layers), but with varying amounts of Dutch and other languages in their pre-training data. We train w2v2-nl ( Wav2Vec2-NL ) on 960 hours of spoken Dutch, combining data from the Spoken Dutch Corpus (CGN; [ 16 ] ), Multilingual Librispeech (MLS; [ 17 ] ), and CommonVoice (CV; [ 18 ] ). We sample 537 hours from CGN, including both Dutch and Flemish recordings but excluding telephone conversations (recordings with low sample rate) and sermons (poor recording quality). The remaining data includes recordings of spontaneous conversations and interviews, as well as read speech and news reports. We segment the CGN recordings into phrases, and use segments between 2 and 15 seconds in length as inputs for model training. From MLS, we sample 211 hours of the provided audiobook segments. The remaining 212 hours are sampled from CommonVoice, consisting of short segments of read aloud speech. Across the full training set, audio samples range between 2 and 32 seconds in length. We follow the original configuration for training Wav2Vec2 in [ 19 ] , only modifying it to allow longer utterance length and per-device batch size. We use the fairseq toolkit [ 20 ] to pre-train w2v2-nl for 100k training steps using 4 Nvidia A100-40GB GPUs. In addition to our Dutch model, we include two additional models trained on other languages, as well as a model trained on non-speech acoustics: fb-en , the self-supervised base model from the original Wav2Vec2 release, pre-trained on 960 hours of English audiobook recordings from LibriSpeech [ 19 ] ; fb-voxp-100k , a multilingual base model pre-trained on 100k hours of European parliament recordings in 23 languages, including 4,5k hours of Dutch [ 21 ] ; and a nonspeech base model trained on 600 hours of acoustic scenes from AudioSet [ 10 ] 3 3 3 HuggingFace [ 22 ] identifiers: amsterdamNLP/Wav2Vec2-NL (w2v2-nl); facebook/wav2vec2-base (fb-en); facebook/ wav2vec2-base-100k-voxpopuli (fb-voxp-100k); ewandunbar/humanlike-speech-2022 (nonspeech) . 3 Representational analysis methods We create the SSL-NL evaluation set by sampling speech recordings from two different datasets of spoken Dutch. We use a separate subset of MLS audiobook segments (held-out from w2v2-nl training data), as well as the IFADV corpus [ 23 ] of face-to-face conversational speech. Based on