Emergent morpho-phonological representations in self-supervised speech models

Jon Gauthier, Canaan Breiss, Matthew Leonard, Edward F. Chang · 2025

Read Emergent morpho-phonological representations in self-supervised speech models by Jon Gauthier, Canaan Breiss, Matthew Leonard, Edward F. Chang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22973.txt.

Introduction

Speech models learn to pick out words from noisy sound, but what codes do they use inside? Jon Gauthier and 3 co-authors probe self-supervised speech models on English word forms. They focus on common noun and verb endings, like plural and past tense forms. The models were tuned for word recognition in natural, messy sound.

Their internal codes show a clean linear geometry. That geometry links base words to their regular changed forms. It does not map neatly onto sound units or grammar units, though. It tracks how word pairs relate across the whole word stock. The result hints that human word recognition may not need split codes for sound and grammar.

Catalog record

CREATORJon Gauthier, Canaan Breiss, Matthew Leonard, Edward F. Chang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-22973

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Emergent morpho-phonological representations in self-supervised speech models Jon Gauthier, Canaan Breiss, Matthew Leonard, Edward F. Chang Abstract Self-supervised speech models can be trained to efficiently recognize spoken words in naturalistic, noisy environments. However, we do not understand the types of linguistic representations these models use to accomplish this task. To address this question, we study how S3M variants optimized for word recognition represent phonological and morphological phenomena in frequent English noun and verb inflections. We find that their representations exhibit a global linear geometry which can be used to link English nouns and verbs to their regular inflected forms. This geometric structure does not directly track phonological or morphological units. Instead, it tracks the regular distributional relationships linking many word pairs in the English lexicon—often, but not always, due to morphological inflection. These findings point to candidate representational strategies that may support human spoken word recognition, challenging the presumed necessity of distinct linguistic representations of phonology and morphology. 1 Introduction In everyday speech perception, humans transform a sequence of rapidly articulated words, often at a rate of several words per second, into coherent meanings. † † Code to reproduce all results is available at https://github.com/hans/nnvb-public . Traditional psycholinguistic models attempt to explain this behavior by positing distinct representations of perceived speech at several linguistic levels: listeners processing individual phonetic segments in a word, along with the phonological, morphological, and semantic properties of a word. This transduction from low- to high-level representation happens through a series of recurrent computations ( McClelland and Elman 1986 ; Gaskell and Marslen-Wilson 1997 ) . While these models successfully explain a host of phenomena within the psycholinguistics of word recognition, they operate at a scale far below today’s self-supervised speech models (S3Ms). Modern speech recognition algorithms that use S3Ms can make context-sensitive transcriptions of noisy acoustic input with an accuracy, efficiency, and scale not achievable with these psychologically motivated models. It is thus necessary to reconcile the theoretical commitments of earlier psycholinguistic models with the behavioral superiority of S3Ms. Recent work has begun to study the representations of S3Ms trained on unlabeled speech data, asking to what extent they recover traditional psycholinguistic notions in phonetics, phonology, morphology, and syntax ( Dunbar et al. 2022 ; Pasad et al. 2023b ; Pasad et al. 2023a ; Martin et al. 2023 ; Sanabria et al. 2023 ; Choi et al. 2024 ; Choi et al. 2025 ) . As a psycholinguistic account, this work departs from prior approaches by searching for model capacities which emerge as a solution to a training objective, rather than seeking to build in hypothesized structures. However, it is not clear how these capacities of S3Ms relate—if at all—to the function of word recognition. Because S3Ms are trained on broad objectives unrelated to any specific linguistic task, their internal states plausibly serve multiple functions beyond recognizing words. As such, we do not know which components of their learned representations are necessary for recognizing words, as opposed to predicting other aspects of the speech signal (from speaker identity and prosody to the identities of individual speech segments). We present an analysis method and a series of controlled experiments on S3Ms, separating representations which are necessary for word recognition from those which emerge in service of more general objectives. We first design a probing method to operationalize the notion of optimal word recognition. This probe targets a subspace of an S3M which serves to distinguish spoken words. In a series of experiments studying the activations within this word-optimal subspace, we discover a highly regular representation of speech sounds. We argue that this representation defies the cleanly separated levels of prior psycholinguistic models, cross-cutting morphological and simpler phonological distinctions, and instead tracks a higher-level pattern that unites multiple morphological inflections that all conform to a single phonological distribution rule. Our findings motivate a new candidate representational strategy that may serve spoken word recognition in human listeners. 2 Background Like many languages, English exhibits regular sound patterns which reflect an interaction of phonological, morphological, and lexical constraints. For example, English words ending in [z], [s], or [Iz] can reflect plural nouns (NNS; e.g. dogs , cats ) and third-person singular verbs (VBZ; e.g. runs , barks ). Both of these inflections introduce a morpheme with the underlying phonological content /-z/ at the right edge of their base, which is realized as one of three surface forms ( allomorphs ) depending on a simple set of distributional constraints, given in 1 . /z/ \rightarrow [z] after voiced non-sibilant sounds ( dogs [dOg z ], runs [⁢rn z ]) /z/ \rightarrow [s] after voiceless non-sibilant sounds ( cats [kæt s ], jumps [dZmp s ]) /z/ \rightarrow [Iz] after sibilant sounds ( dishes [dIS Iz ], finishes [fInIS Iz ]) List of myboxs 1 Distributional constraints describing how the word-final /-z/ of English noun plurals and third-person verb inflections are realized in different surface forms [z], [s], and [Iz]. Of course, not all word-final instances of [z], [s], and [Iz] are generated by these morpho-phonological processes: consider monomorphemic words such as haze , fleece , six , and hearse . Some of these words happen to end in sequences that are consistent with the constraints of 1 , while some do not: haze and six end in a sound matching the voicing of the preceding sound (consistent); fleece and hearse end in sounds which do not match the voicing of the preceding sound (inconsistent). A combination of morphological, phonological, and lexical processes produce these word-final sounds. We exploit these multiple levels of patterning in a series of experiments on a model of word recognition, asking how representations at these levels are negotiated by the model in service of its objective. 3 Methods We design experiments targeting four (non-exclusive) hypotheses about the linguistic representations in self-supervised speech models. From model activations computed on the word-final sounds [z], [s], and [Iz], our experiments evaluate: Morphological sensitivity (§ 4.1 ): A morphologically sensitive representation of [z] would contrast instances of the plural [z] (e.g. in daughters ) from instances of third-person singular [z] (in enters ); likewise for [s] and [Iz]. Phonological sensitivity (§ 4.2 ): A phonologically sensitive representation of [z] would contrast instances where it surfaces as [z] (e.g. daughters ) from instances where it surfaces as [s] and [Iz] (e.g. lips and cheeses ); and likewise for [s] and [Iz] in both noun plurals in verb inflections. Lexical sensitivity (§ 4.3 ): A lexically sensitive representation of [z] would contrast instances where the [z] is part of a lexical form (e.g. haze ) from instances where it is inflectional (e.g. daughters ); and likewise for [s] and [Iz]. Distributional sensitivity (§ 4.4 ): A representation sensitive to the distributional constraints given in 1 would contrast sounds [z], [s] and [Iz] in contexts where they are consistent with these rules (e.g. haze ) from contexts where they are inconsistent (e.g. fleece ). We address these questions using two types of models: a self-supervised speech model trained with a general contrastive learning objective, and a fine-tuned variant trained specifically for word recognition. By comparing answers to the above questions across these two models, we can assess which levels of linguistic representation

Read the complete text