Tracking the emergence of linguistic structure in self-supervised models learning from speech

Marianne de Heer Kloots, Martijn Bentum, Hosein Mohebbi, Charlotte Pouw, Gaofei Shen, Willem Zuidema, [0.5em] Institute for Logic, Language, Computation, University of Amsterdam, The Netherlands · 2026

Read Tracking the emergence of linguistic structure in self-supervised models learning from speech by Marianne de Heer Kloots, Martijn Bentum, Hosein Mohebbi, Charlotte Pouw, Gaofei Shen, Willem Zuidema, [0.5em] Institute for Logic, Language, Computation, University of Amsterdam, The Netherlands (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-02043.txt.

Description

de Heer Kloots and colleagues track when linguistic structure emerges in speech models trained on spoken Dutch. Self supervised models learn linguistic structure, but when it emerges in training was unclear. Studying layers and checkpoints across six Wav2Vec2 and HuBERT models, they find distinct layer patterns and learning paths. Those differences partly reflect abstraction from sound and input timescale. Pretraining objectives shape layer organization and learning paths, with higher order tasks using iteratively refined pseudo labels inducing greater parallelism.

Catalog record

CREATORMarianne de Heer Kloots, Martijn Bentum, Hosein Mohebbi, Charlotte Pouw, Gaofei Shen, Willem Zuidema, [0.5em] Institute for Logic, Language, Computation, University of Amsterdam, The Netherlands
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47332
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.02043

Browse the full library · About the Tower of Babel