Tracking the emergence of linguistic structure in self-supervised models learning from speech
Marianne de Heer Kloots, Martijn Bentum, Hosein Mohebbi, Charlotte Pouw, Gaofei Shen, Willem Zuidema, [0.5em] Institute for Logic, Language, Computation, University of Amsterdam, The Netherlands · 2026
Read Tracking the emergence of linguistic structure in self-supervised models learning from speech by Marianne de Heer Kloots, Martijn Bentum, Hosein Mohebbi, Charlotte Pouw, Gaofei Shen, Willem Zuidema, [0.5em] Institute for Logic, Language, Computation, University of Amsterdam, The Netherlands (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-02043.txt.
Description
de Heer Kloots and colleagues track when linguistic structure emerges in speech models trained on spoken Dutch. Self supervised models learn linguistic structure, but when it emerges in training was unclear. Studying layers and checkpoints across six Wav2Vec2 and HuBERT models, they find distinct layer patterns and learning paths. Those differences partly reflect abstraction from sound and input timescale. Pretraining objectives shape layer organization and learning paths, with higher order tasks using iteratively refined pseudo labels inducing greater parallelism.
Catalog record
| CREATOR | Marianne de Heer Kloots, Martijn Bentum, Hosein Mohebbi, Charlotte Pouw, Gaofei Shen, Willem Zuidema, [0.5em] Institute for Logic, Language, Computation, University of Amsterdam, The Netherlands |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47332 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.02043