Normalization through Fine-tuning: Understanding Wav2vec 2.0 Embeddings for Phonetic Analysis

Yiming Wang, Yi Yang, Jiahong Yuan · 2025

Read Normalization through Fine-tuning: Understanding Wav2vec 2.0 Embeddings for Phonetic Analysis by Yiming Wang, Yi Yang, Jiahong Yuan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04814.txt.

Introduction

Yiming Wang, Yi Yang, and Jiahong Yuan study normalization inside wav2vec 2.0 embeddings in two thousand twenty-five. Speech work needs features that compare fairly across speakers. Classic pipelines normalized sounds as a fixed first step. Fine tuned transformer models fold that step inside, out of sight.

The team probes embeddings from models tuned for varied tasks. Fine tuning does achieve phonetic normalization, they find. It works by muting facts that the task does not need. Models tuned for two tasks kept facts for both with no loss. Muting also proved needless for sound classification itself. The results mirror how human hearing may flex.

Catalog record

CREATORYiming Wang, Yi Yang, Jiahong Yuan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-04814

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Normalization through Fine-tuning: Understanding Wav2vec 2.0 Embeddings for Phonetic Analysis Yiming Wang, Yi Yang, Jiahong Yuan Abstract Phonetic normalization plays a crucial role in speech recognition and analysis, ensuring the comparability of features derived from raw audio data. However, in the current paradigm of fine-tuning pre-trained large transformer models, phonetic normalization is not deemed a necessary step; instead, it is implicitly executed within the models. This study investigates the normalization process within transformer models, especially wav2vec 2.0. Through a comprehensive analysis of embeddings from models fine-tuned for various tasks, our results demonstrate that fine-tuning wav2vec 2.0 effectively achieves phonetic normalization by selectively suppressing task-irrelevant information. We found that models fine-tuned for multiple tasks retain information for both tasks without compromising performance, and that suppressing task-irrelevant information is not necessary for effective classification. These findings provide new insights into how phonetic normalization can be flexibly achieved in speech models and how it is realized in human speech perception. I Introduction Transformer-based speech models, such as wav2vec 2.0, have shown remarkable success in phonetics-related tasks, including phoneme recognition, tone recognition, and speaker identification. To leverage wav2vec 2.0 embeddings for phonetic analysis, it is essential to understand how these embeddings encode information, particularly with regard to phonetic normalization. For example, when analyzing the acoustic differences among lexical tones to uncover inherent tonal features in a language, speaker-specific variations, e.g., females have a higher pitch than males, must be normalized to allow for meaningful comparisons of tonal features across speakers. Probing experiments have demonstrated that the embeddings of pre-trained wav2vec 2.0 models encode various types of information, with different layers capturing different aspects. Our study takes this a step further by investigating whether and how fine-tuning wav2vec 2.0 models for a specific task can suppress information irrelevant to that task. Moreover, we examine how the embeddings differ when the model is fine-tuned for multiple tasks compared to a single task. Our main findings are summarized as follows: • Phonetic normalization is achieved through the fine-tuning of wav2vec 2.0. The process, from the initial to the final layers of a fine-tuned Transformer, varies depending on the type of information: for tones and finals, the information is initially enhanced and subsequently suppressed, while for speaker sex, the information is progressively suppressed throught the layers. The observed differences can likely be attributed to the way information is encoded across the various layers of the pretrained wav2vec 2.0 model. • The embeddings from a model fine-tuned for multiple tasks, such as tone and sex classification, effectively retain information of both tone and sex. The effectiveness of these embeddings is comparable to those fine-tuned for each task separately. • All task-irrelevant information is suppressed in fine-tuned models. For instance, the embeddings of a model fine-tined for tone recognition suppresses both sex and final information. This is not an obvious outcome from a machine learning perspective, as embeddings containing both tone and sex information perform equally well for optimizing the objective of tone classification. In other words, when fine-tuning for tone classification, suppressing sex information is not strictly necessary. • The combination of results 2 and 3 have an important implication for phonetic analysis. Phonetic normalization does not necessarily require the suppression of task-irrelevant information. For example, sex or final information can be either suppressed or preserved in features that effectively distinguish tones. This finding provides new insights into how phonetic normalization can be flexibly achieved in speech models and how it is realized in human speech perception. II Related work II-A Normalization Normalization in human speech perception has long been studied [ 1 ] . [ 2 ] investigated the effect of F0 on the perception of vowels using synthetic sounds, showing that the perceptual category boundary between vowels shifted when the F0 was doubled. [ 3 ] found that listeners can determine where the pitch of an isolated vowel, produced by a stranger, falls within that stranger’s pitch range, without prior familiarity or context. [ 4 ] used high-density electrocorticography to record neural population activity while participants listened to sentences that varied in intonational pitch contour, phonetic content, and speaker. They found that the cortical representation of intonation relies on relative-pitch encoding, rather than absolute-pitch encoding. In phonetic research, numerous vowel normalization methods have been proposed in the literature [ 5 , 6 ] . Regarding pitch and tone normalization, commonly utilized techniques include z-scores, semitones, and the 5-point scale [ 7 , 8 ] . In speech technology, techniques such as speaker adaptive training (SAT), vocal tract length normalization (VTLN), and Maximum Likelihood Linear Regression (MLLR) are extensively employed for speaker normalization and adaptation [ 9 ] . In deep learning, a variety of normalization techniques have been developed to speed up training and mitigate issues related to vanishing and exploding gradients. These include batch normalization, layer normalization, weight normalization, and spectral normalization, among others. It’s important to note that these methods do not incorporate explicit speaker or phonetic normalization as discussed above. II-B wav2vec 2.0 wav2vec 2.0 is a self-supervised learning framework based on Transformers that can learn speech representations from raw audio data [ 10 ] . The framework processes the speech signal with a multilayer convolutional network to extract latent features, which are then fed into vector quantization and transformer networks. It is pre-trained on a noise contrastive binary classification task, which enables the model to capture a rich amount of information about speech, as demonstrated by probing experiments that show their effectiveness on a wide range of tasks [ 11 , 12 , 13 ] . The pre-trained models of wav2vec2 can be fine-tuned for speech recognition using labeled data [ 10 ] . [ 14 ] fine-tuned wav2vec 2.0 with a CTC loss to recognize suprasegmentals such as syllables, tones, and pitch accents. Their results demonstrated a 50% error reduction in Mandarin tone recognition compared to previous studies. [ 15 ] proposed fine-tuning wav2vec 2.0 with a cross-entropy loss to classify tones in an utterance on a frame-by-frame basis, which not only improves tone classification accuracy but also generates frame-level representations suitable for tonal analysis. Using features extracted from a pre-trained wav2vec 2.0 model for phoneme recognition, [ 16 ] discovered that the features from the first 10 layers as well as the last few layers provide very poor performance, while layers 15-19 achieve the best phoneme recognition accuracy. In this study, we will investigate the performance of features extracted from various layers of fine-tuned wav2vec 2.0 models across multiple tasks. III Data and method III-A Data Our experiments were conducted on the Aishell-1 dataset [ 17 ] , which is widely used for Mandarin ASR. It consists of 165 hours of read speech and word transcriptions in Mandarin Chinese from 400 speakers. The dataset is divided into train, dev, and test sets with 150, 10, and 5 hours of speech, respectively. III-B Forced Alignment We trained an HMM-GMM based forced aligner on the dataset using the HTK toolkit 1 1 1 https://htk.eng.cam.ac.uk/ and the pronouncing dictionary included in the dataset. The dictionary transcribes words into initials and tonal finals in Pinyin, a Roman alphabet system of phonetic transcription for Mandarin Chinese. From forced alignment, we assigned a label to each frame

Read the complete text