Revealing the Hidden Temporal Structure of HubertSoft Embeddings based on the Russian Phonetic Corpus
Anastasia Ananeva, Anton Tomilov, Marina Volkova · 2025
Read Revealing the Hidden Temporal Structure of HubertSoft Embeddings based on the Russian Phonetic Corpus by Anastasia Ananeva, Anton Tomilov, Marina Volkova (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06794.txt.
Introduction
Anastasia Ananeva, Anton Tomilov, and Marina Volkova report a study from two thousand twenty-five on hidden time structure inside speech model embeddings. Self-taught models like HuBERT read phonetic detail from raw audio without labels. Whether their outputs also keep the order of sounds over time was an open question.
The team used a Russian speech corpus and a soft-clustering variant of the model. They tagged brief twenty-millisecond windows at sound boundaries with triplets naming the sounds before, at, and after each point. A small network then tried to predict each position from the embedding alone. It read identity and order well, best at the boundaries themselves. Errors hinted that the embeddings also carry fine details of how sounds blend into each other.
Catalog record
| CREATOR | Anastasia Ananeva, Anton Tomilov, Marina Volkova |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-06794
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revealing the Hidden Temporal Structure of HubertSoft Embeddings based on the Russian Phonetic Corpus Thanks: Corresponding author Anastasia Ananeva, Anton Tomilov, Marina Volkova Abstract Self-supervised learning (SSL) models such as Wav2Vec 2.0 and HuBERT have shown remarkable success in extracting phonetic information from raw audio without labelled data. While prior work has demonstrated that SSL embeddings encode phonetic features at the frame level, it remains unclear whether these models preserve temporal structure, specifically, whether embeddings at phoneme boundaries reflect the identity and order of adjacent phonemes. This study investigates the extent to which boundary-sensitive embeddings from HubertSoft, a soft-clustering variant of HuBERT, encode phoneme transitions. Using the CORPRES Russian speech corpus, we labelled 20 ms embedding windows with triplets of phonemes corresponding to their start, centre, and end segments. A neural network was trained to predict these positions separately, and multiple evaluation metrics, such as ordered, unordered accuracy and a flexible centre accuracy, were used to assess temporal sensitivity. Results show that embeddings extracted at phoneme boundaries capture both phoneme identity and temporal order, with especially high accuracy at segment boundaries. Confusion patterns further suggest that the model encodes articulatory detail and coarticulatory effects. These findings contribute to our understanding of the internal structure of SSL speech representations and their potential for phonological analysis and fine-grained transcription tasks. 1 Introduction Self-supervised learning (SSL) models such as Wav2Vec 2.0 [ 1 ] and HuBERT [ 2 ] have significantly advanced the field of speech representation learning by extracting rich phonetic information without labelled data. The embeddings received usually contain a large amount of information which is used in deepfake detection [ 3 ] . These models have demonstrated impressive performance on tasks including phoneme classification, automatic speech recognition and articulatory feature prediction. However, it remains underexplored how these embeddings encode phoneme boundaries and whether the temporal structure is also encoded. It has been shown that SSL embeddings encode temporally localized phonetic cues accessible through supervised probing [ 4 ] . For instance, linear or shallow MLP probes have effectively predicted phonetic features such as place, manner, and voicing at the frame level [ 5 , 6 ] . These works use Wav2Vec 2.0 to confirm that transitions in the embedding space align closely with phoneme boundaries, offering indirect evidence of temporal structure encoding. Other approaches track articulatory feature trajectories or analyse embedding differences at phoneme transitions, further supporting the presence of boundary-sensitive representations [ 7 ] . Phonetic features are also reflected in the embeddings received with HuBERT [ 10 , 11 ] . However, these studies focus mainly on some phonetic features encoded in the embeddings. For example, HuBERT embeddings differentiate vowel quality by encoding distinctions in vowel height, backness, and roundedness. They also effectively represent consonantal articulation features such as place and manner, voicing, nasality, and the stop–fricative contrast. Beyond segmental properties, HuBERT embeddings have been found to reflect prosodic features like pitch, stress, and intonation, and are sensitive to phonetic context effects, including coarticulation. In addition, there is growing evidence that both self-supervised and supervised models can be effectively used for phoneme boundary detection. Self-supervised models have been shown to capture abrupt spectral or articulatory transitions in their embeddings, which align closely with phoneme boundaries [ 8 ] . Techniques such as comparing adjacent frame representations or applying peak detection over embedding dissimilarities allow for boundary identification without explicit labels, suggesting that SSL models implicitly encode segmental structure alongside phonetic content. Complementing this, supervised approaches such as those using Connectionist Temporal Classification (CTC) loss have framed the task as aligning sequences of phoneme pairs to speech. This enables models to focus on transition regions and predict boundaries with high temporal precision [ 9 ] . Together, these lines of work highlight the capacity of learned representations to reflect temporal dynamics in speech. Despite this progress, to the best of our knowledge, no prior study has directly tested whether SSL embeddings encode the identity of the phoneme that begins or ends a segment encoded. The existing literature focuses predominantly on general framewise phonetic classification or the analysis of internal embedding structure. This gap is especially pertinent given the growing interest in understanding how SSL models internalize sub-segmental linguistic structure. Notably, there is also a lack of probing studies involving HubertSoft [ 12 ] , a modification of HuBERT which generates soft posterior probabilities over learned acoustic units instead of hard cluster assignments. The current work aims to fill this gap by evaluating whether boundary-frame embeddings from HubertSoft can be used to predict the phoneme that begins or ends a segment. By focusing on supervised probing of initial and final frames at phoneme boundaries, we offer a novel perspective on the structure encoded within self-supervised speech embeddings and shed light on the representations’ utility for fine-grained phonological tasks. 2 Material The experiments were conducted using the CORPRES [ 13 ] Russian speech corpus. The dataset contains recordings of various speakers, annotated on multiple linguistic levels. Phonetic labeling was performed by trained phoneticians, ensuring a high degree of reliability. However, in certain instances, particularly at the boundaries between acoustically similar sounds, such as between two vowels or a vowel and a consonant, the segmentation may be imprecise. In such cases, the boundary is typically placed approximately at the midpoint of the acoustic transition between the sounds [ 14 ] . The hand-labelled recordings from eight speakers were utilized for embedding extraction with the HubertSoft model. For embedding extraction, the HubertSoft model was used with frozen weights, while only the classification heads were trained. Initially, embeddings were extracted from the recordings and then averaged within phonetic segment boundaries, providing a compact representation of each labelled sound. These averaged embeddings were used in a subset of experiments to establish baseline performance. The symbols for various sounds were also taken from the CORPRES. Figure 1: Example of embedding extraction and a triplet label generation based on manual phonetic annotation across 20ms time frames. To move towards more fine-grained, automatic analysis suitable for speech transcription tasks, we also extracted non-averaged embeddings. In this setup, each embedding represented a 20 ms window of the audio signal, as shown in Figure 1 , without overlap. This approach preserved high temporal resolution, which was essential due to the presence of very short phonetic events (as brief as 5 ms) in the corpus. Such short phoneme realizations are usually unstressed vowels or /r/ and they appear due to the fluency of speech. They may not be pronounced properly, but still are labelled to reflect the phonetic composition of speech. Each embedding was assigned a triplet label indicating the phoneme present at its beginning, centre, and end, derived from the time-aligned phonetic annotations. Table 1 gives some examples of possible labels. This labelling scheme allowed us to capture phonetic transitions within the window. In most cases, all three sub-labels were identical, suggesting that the window was fully inside a single phoneme (typically its central part). However, a substantial number of embeddings spanned phoneme boundaries, where the centre label often matched either the start or end label. These boundary-spanning embeddings are particularly informative for analysing whether the learned representations reflect the temporal structure of the speech signal. Table 1: Examples of