Iterative refinement, not training objective, makes HuBERT behave differently from wav2vec 2.0

Robin, Ewan · 2025

Read Iterative refinement, not training objective, makes HuBERT behave differently from wav2vec 2.0 by Robin, Ewan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08110.txt.

Introduction

Robin Huo and Ewan Dunbar compare two famed speech models to find why they learn speech so well. Both models learn from raw sound with no labels, yet their inner maps of words, sounds, and voices differ. The pair differ in two ways: the goal used in training, and a loop that refines fake labels over rounds. Their 2025 study splits those two traits apart in clean tests.

Round-by-round refinement, not the training goal, turns out to drive the gap between the models. Maps from refined models line up far more with word, sound, and speaker facts. The authors urge more work on why the loop works so well. The finding can steer how next speech models are built and trained.

Catalog record

CREATORRobin, Ewan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-08110

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Iterative refinement, not training objective, makes HuBERT behave differently from wav2vec 2.0 Robin, Ewan Abstract Self-supervised models for speech representation learning now see widespread use for their versatility and performance on downstream tasks, but the effect of model architecture on the linguistic information learned in their representations remains under-studied. This study investigates two such models, HuBERT and wav2vec 2.0, and minimally compares two of their architectural differences: training objective and iterative pseudo-label refinement through multiple training iterations. We find that differences in canonical correlation of hidden representations to word identity, phoneme identity, and speaker identity are explained by training iteration, not training objective. We suggest that future work investigate the reason for the effectiveness of iterative refinement in encoding linguistic information in self-supervised speech representations. Huo Dunbar ]Department of LinguisticsUniversity of TorontoCanada ]Department of Computer ScienceUniversity of TorontoCanada ]Department of FrenchUniversity of TorontoCanada keywords speech representations, HuBERT, wav2vec 2.0, iterative refinement, self-supervised learning † † email: [email protected], [email protected] 1 Introduction The use of self-supervised learning (SSL) methods for learning speech representations has become commonplace in recent years, owing to its improvement of downstream performance, reduction of the amount of labelled data needed to develop task-specific models, and versatility in a variety of applications. However, while the utility of SSL representations for downstream tasks has been well shown [ 1 ] , and while it is understood that many forms of speech SSL learn meaningful linguistic representations [ 2 , 3 , 4 ] , the reasons why they succeed have remained unclear. In particular, the effect of choices in model architecture and training regime on the linguistic information and structure in their representations remains under-studied. Of the SSL models for speech in popular use today, most are based on the architecture of HuBERT [ 5 ] or wav2vec 2.0 [ 6 ] . Recent work has shown that these two models show appreciable differences with respect to the linguistic information in their learned representations [ 3 , 4 , 7 , 8 ] . Despite these behavioural differences, HuBERT and wav2vec 2.0 have only a few important differences in architecture. First, while both models predict pseudo-labels of masked input frames, HuBERT optimizes a masked language modelling classification objective, whereas wav2vec 2.0 optimizes a contrastive objective with negative examples. Second, HuBERT obtains the pseudo-labels for this task from k-means clustering on acoustic features or existing HuBERT representations and keeps them fixed, whereas wav2vec 2.0 jointly learns its pseudo-labels using a quantization module. Finally, HuBERT is pretrained in multiple iterations with each trained iteration providing representations to be clustered into the pseudo-label categories of the next, while wav2vec 2.0 is pretrained only once. We isolate and investigate the effect of the training objective and iterative refinement of pseudo-labels via multiple training passes on the linguistic information encoded in the resulting pretrained model. We find that the critical difference is the use of iterative refinement. The behaviour of a HuBERT-like model with respect to layer-wise correlation of its representations to words and phonemes is predicted by the number of training iterations, not by training objective: as the training iteration increases, so does the level of linguistic correlation in the final layers. Furthermore, this improvement in linguistic encoding across iterations appears to come at the cost of non-linguistic speaker information. On the basis of these results, we suggest that future research investigate the reason for the effectiveness of iterative refinement in producing linguistically meaningful representations, and develop ways to leverage this insight for more efficient training of high-quality representation learners. Code and checkpoints for our investigation can be found at https://github.com/RobinHuo/iter-ref . 2 Preliminaries 2.1 HuBERT and wav2vec 2.0 HuBERT [ 5 ] and wav2vec 2.0 [ 6 ] are popular SSL speech models which have inspired many subsequent variants and architectures [ 9 , 10 , 11 , for example] . The base variants of HuBERT and wav2vec 2.0 consist of a 7-layer convolutional waveform encoder, followed by a 12-layer bidirectional Transformer encoder [ 12 ] , and a final projection. In pretraining, both models use a masked prediction task with a loss of the form \mathcal{L}_{\text{masked}}=-\log\frac{\exp(\operatorname{sim}(x,e_{c})/\tau)}{\sum_{c^{\prime}\in C}\exp(\operatorname{sim}(x,e_{c^{\prime}})/\tau)} where \operatorname{sim}(\cdot,\cdot) is cosine similarity, c is the pseudo-label of the masked frame being predicted, x is the output of the model for that frame, e_{p} is the embedding for pseudo-label p , \tau is a temperature hyperparameter, and C is the set of pseudo-labels from which the model must predict the correct choice. The set C differs between HuBERT and wav2vec 2.0: in HuBERT this is the set of all pseudo-labels, whereas in wav2vec 2.0 it is a set consisting of the correct pseudo-label c along with K negative examples sampled from the pseudo-labels of other masked frames in the same utterance, where K is a hyperparameter. The loss used by wav2vec 2.0 is termed a contrastive loss, as it is designed to make the model learn to distinguish between frames in the same utterance. The HuBERT loss is termed predictive. The pseudo-labels used by HuBERT and wav2vec 2.0 are computed differently. In HuBERT, the pseudo-labels remain fixed for the duration of training, are subsequently updated, and training is then restarted with a newly initialized model (iterative refinement). The first training iteration uses pseudo-labels from clustered acoustic features of the input while subsequent iterations use pseudo-labels from clustered hidden representations yielded by the previous training iteration. In contrast, wav2vec 2.0 undergoes only one training pass using an online quantization module (2-codebook product quantization learned with Gumbel softmax) to jointly learn a set of pseudo-labels. In order to ensure full use of this quantization module, wav2vec 2.0 also employs a diversity loss in addition to the masked prediction loss ( \mathcal{L}=\mathcal{L}_{\text{masked}}+\alpha\mathcal{L}_{\text{diversity}} , where \alpha is a hyperparameter), which encourages equal use of all codewords. 2.2 Canonical correlation analysis We follow Pasad et al. [ 13 , 3 , 4 ] in using canonical correlation analysis (CCA) to evaluate the linguistic information contained in SSL representations. CCA is a technique for characterizing the relationship between two random vectors in terms of correlations between linear combinations of their features. Given n pairs of vectors (x,y) sampled from the random vectors X\in\mathbb{R}^{d_{x}} and Y\in\mathbb{R}^{d_{y}} , CCA computes \min\{d_{x},d_{y}\} canonical variable pairs U_{i}=u_{i}^{\mathsf{T}}X and V_{i}=v_{i}^{\mathsf{T}}Y where u_{i}\in\mathbb{R}^{d_{x}} and v_{i}\in\mathbb{R}^{d_{y}} , such that the Pearson correlation \operatorname{cor}\left(U_{i},V_{i}\right) is maximized for each i subject to the constraint that each canonical variable U_{i} and V_{i} is uncorrelated with both U_{j} and V_{j} for all j<i . The quantity \operatorname{cor}\left(U_{i},V_{i}\right) is the i -th canonical correlation. Projection-weighted CCA [ 14 ] is a variant of CCA which returns a weighted sum of the canonical correlations such that directions accounting for higher proportion of the input receive higher weight. Intuitively, the scalar value returned by projection-weighted CCA is an integrated measure of similarity, as measured by canonical correlations, between two multivariate data series of interest. When comparing model representations to linguistic categories using this method, we henceforth refer to this value as the CCA score. 3 Influence of model architecture and training on linguistic encoding We investigate the effect of specific factors in self-supervised speech model architecture and training on the linguistic information encoded in the hidden representations of the resulting pretrained model, focusing

Read the complete text