Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces

Kwanghee, Eunjung, Cheol Jun, David R, David · 2026

Read Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces by Kwanghee, Eunjung, Cheol Jun, David R, David (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12642.txt.

Introduction

Kwanghee Choi and co-authors probe how self-supervised speech models pack phonetic context into a single frame. Prior work showed that traits of one phone, like voicing, sit superposed in its vector. The team argues that traits of nearby phones are superposed there too, split by relative position. They find that position subspaces are near orthogonal to each other. Implicit phonetic boundaries also emerge from this structure. The findings sharpen what contextualized means for speech models.

Catalog record

CREATORKwanghee, Eunjung, Cheol Jun, David R, David
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39978
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.12642

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Self-Supervised Speech Models Encode Phonetic Context via Position-dependent Orthogonal Subspaces Kwanghee, Eunjung, Cheol Jun, David R, David Abstract Transformer-based self-supervised speech models (S3Ms) are often described as contextualized, yet what this entails remains unclear. Here, we focus on how a single frame-level S3M representation can encode phones and their surrounding context. Prior work has shown that S3Ms represent phones compositionally; for example, phonological vectors such as voicing, bilabiality, and nasality vectors are superposed in the S3M representation of [m]. We extend this view by proposing that phonological information from a sequence of neighboring phones is also compositionally encoded in a single frame, such that vectors corresponding to previous, current, and next phones are superposed within a single frame-level representation. We show that this structure has several properties, including orthogonality between relative positions, and emergence of implicit phonetic boundaries. Together, our findings advance our understanding of context-dependent S3M representations. 1 1 1 All code for experiments is available at https://anonymous.4open.science/r/phonetic-arithmetic-1CF2 . Choi Yeo Cho Mortensen Harwath keywords self-supervised speech models, interpretability, phonological features, phonetic context, contextualization † † address: 1 UT Austin 2 UC Berkeley 3 Carnegie Mellon University † † email: {kwanghee,harwath}@utexas.edu 1 Introduction Self-supervised speech models (S3Ms) have become a central paradigm for learning general-purpose speech representations from unlabeled audio, enabling strong performance across a wide range of downstream tasks [ 3 , 27 , 8 ] . The neural architecture of this class of models was introduced by wav2vec [ 44 ] and has since been widely adopted by subsequent S3Ms [ 3 , 27 , 8 ] . The architecture consists of two main components: a convolutional network that transforms the raw waveform into a sequence of latent representations, and a transformer-based context network that can combine information across time. While this architecture has proven highly effective, its internal mechanisms remain only partially understood. Prior work has identified what kinds of acoustic and linguistic information is present in different layers of S3Ms [ 41 , 42 ] . However, despite covering multiple levels of the linguistic hierarchy, it remains unclear how the context network organizes such information within the representation space. This gap has motivated a growing body of work aimed at characterizing the internal structure of representations learned by S3Ms [ 33 , 36 , 20 , 16 , 19 , 18 , 48 , 50 ] . (a) Conceptual illustration of our hypothesis. (b) Empirical demonstration of our hypothesis. Figure 1 : (a) Each frame-level S3M representation contains phonological vectors from position-sensitive, orthogonal subspaces. (b) Cosine similarity between frame-level representations and phonological vectors ( low , strident ) from the previous, current, and next phone subspaces. Individual frames encode phonological information from the current and neighboring phones with respect to phonetic boundaries. [ 41 ] has shown that the convolutional networks of S3Ms primarily capture local acoustic information. Similar to a spectrogram, the limited receptive field enforces temporal locality, with each frame aligned to a specific region of the input signal. Further, [ 20 ] showed that, unlike spectrograms—whose dimensions correspond to fixed frequency bands—the convolutional representation space is distributed and relative. In this space, acoustic similarity is reflected in the cosine similarity between representations rather than along fixed axes. These findings suggest that while convolutional network can provide local acoustic information, they do not by themselves account for how information beyond local acoustics is incorporated within S3Ms. Despite this relatively clear picture concerning the convolutional network, the role of the transformer-based context network is less well understood. Empirically, the context network is crucial. Representations from later transformer layers consistently yield stronger performance on speech probing tasks, such as phone and word identity, compared to convolutional representations [ 42 ] , and combining information across transformer layers often further improves performance [ 47 ] . Moreover, scaling the context network typically leads to substantial gains, with Large models consistently outperforming Base models [ 3 , 27 , 8 ] . These trends suggest that the context network contributes more than a simple refinement of local acoustics. An additional clue comes from the effectiveness of discretized S3M representations in various speech applications [ 7 ] , where k-means clustering of S3M representations yields discrete units that align with phonetic categories despite the absence of supervision [ 50 , 3 , 27 , 21 ] . Moreover, the units tend to organize according to phonetic categories. For example, phones from the same category exhibit higher mutual similarity than phones from different category ( e.g. , vowels vs. consonants) [ 1 , 48 ] . Finally, phonetic and syllabic boundaries have also been observed in attention maps [ 32 ] , enabling the use of S3Ms for unsupervised speech segmentation [ 40 , 2 , 10 , 9 , 49 ] . These observations suggest that context networks encode linguistically meaningful distinctions rather than merely reflecting fine-grained acoustic similarity. To explain the emergence of such linguistically structured representations, recent work has proposed that S3Ms encode phonological vectors [ 19 ] . Under this view, individual phones are represented with linear combinations of phonological vectors, yielding a compositional representation space. For example, the S3M representation of [æ] contains a linear combination of phonological vectors, including low, front, and unrounded vectors, in a way that supports vector arithmetic. While this framework provides a coherent explanation for phonological capabilities described above, it considers only individual phones in isolation and does not examine how phonetic context from neighboring phones are captured by transformers. This motivates us to extend their framework by hypothesizing that each frame-level S3M representation encodes not only a single phone, but sequences of phones ( Figure 1 ). This extension aims to shed light on how S3Ms encode contextual information. Specifically, we propose each contextualized frame-level representation contains a linear combination of phonological vectors corresponding not only to the phone aligned with that frame, but also to its neighboring phones. Based on this hypothesis, we formulate the following four predictions. Prediction 1 ( Section 3.1 ). Individual frame-level representations will exhibit a compositional phonological structure. We assess whether vector arithmetic within the representation space reflects phonological analogies, e.g. , adding a voicing vector [d] - [t] to a voiceless stop [p] approximates the voiced stop [b]. While [ 19 ] rely on mean-pooled representations, we evaluate whether such phonological analogies hold at the level of individual frame representations. Prediction 2 ( Section 3.2 ). Individual frame-level representations will encode phonological information of neighboring phone segments as well. To test this, we extend the above phonological analogy test to neighboring phones, examining whether individual frame representations support phonological analogies on previous or next phones. Concretely, we test whether there exist vectors in the representation space that encode contextual phonological properties such as “is the previous phone voiced?” or “is the next phone a high vowel?”, and whether such properties can be recovered via linear operations on single-frame representations. Prediction 3 ( Section 3.3 ). Position-dependent subspaces will be orthogonal to one another, allowing the model to distinguish which information corresponds to each relative position, e.g. , previous, current, or next phone. For example, if the current phone and the previous phone are voiced, a representation that simply sums shared phonological

Read the complete text