UNSUPERVISED LEARNING AND REPRESENTATION OF MANDARIN TONAL CATEGORIES BY A GENERATIVE CNN

Kai Schenck, Gašper Beguš · 2025

Read UNSUPERVISED LEARNING AND REPRESENTATION OF MANDARIN TONAL CATEGORIES BY A GENERATIVE CNN by Kai Schenck, Gašper Beguš (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-17859.txt.

Introduction

Kai Schenck and Gasper Begus test if a speech model can learn Mandarin tones with no labels at all. Their 2025 paper uses a generative net that makes speech from hidden choice codes. Tones are a hard test, since pitch alone sets word sense apart.

All three trained models tied their choice codes to pitch gaps that mark tones. A model trained on male voices only coded tone in a steady way. The learnt steps look like a stage in how kids pick up tones. The team also maps where tone facts sit inside the net's layers.

Catalog record

CREATORKai Schenck, Gašper Beguš
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-17859

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UNSUPERVISED LEARNING AND REPRESENTATION OF MANDARIN TONAL CATEGORIES BY A GENERATIVE CNN Kai Schenck, Gašper Beguš Abstract This paper outlines the methodology for modeling tonal learning in fully unsupervised models of human language acquisition. Tonal patterns are among the computationally most complex learning objectives in language. We argue that a realistic generative model of human language (ciwGAN) can learn to associate its categorical variables with Mandarin Chinese tonal categories without any labeled data. All three trained models showed statistically significant differences in F0 across categorical variables. The model trained solely on male tokens consistently encoded tone. Our results suggest that not only does the model learn Mandarin tonal contrasts, but it learns a system that corresponds to a stage of acquisition in human language learners. We also outline methodology for tracing tonal representations in internal convolutional layers, which shows that linguistic tools can contribute to interpretability of deep learning and can ultimately be used in neural experiments. † † address: University of California, Berkeley {kai_schenck, begus}@berkeley.edu 1 Introduction Unsupervised deep learning models can address the question of whether language-specific cognitive machinery is required to acquire phonological contrasts from unlabeled audio input. Deep neural networks in particular are general learners, in the sense that they are capable of approximating any continuous function on a bounded domain [ 1 ] . Therefore, if a deep neural network is able to successfully both distinguish and produce phonological contrasts, this would provide evidence that domain-general cognitive processes may also be capable of doing so without language-specific machinery. These models can also serve as (imperfect) proxies of human language acquisition and use, and interpretability techniques can be used to gain a window into their internal representations that is currently out of reach for human brains. Although there are numerous candidate contrasts or processes that could be investigated in this vein, tone – systematic variance in fundamental frequency that encodes lexical or grammatical meaning [ 2 ] – is uniquely suitable for this purpose. While pitch is one of the simplest phonetic properties to measure, tonal alternations are among the most computationally complex phonological processes in human language [ 3 ] . We investigate whether the ciwGAN deep learning model [ 4 ] is able to infer the four contrastive Mandarin tonal categories from unlabeled audio data and produce examples of those categories. We show that the model trained on male-only data is able to successfully map analogues to the Mandarin tonal contrasts to its categorical variables, and that these categorical variables are encoded in some form in the two latest intermediate convolutional layers, but not in earlier layers. The model trained on data of both sexes, meanwhile, is able to partially learn these categories; although it is not able to consistently encode all four categories using its categorical variables, we show that the tonal trajectories it does more consistently encode categories corresponding to the first two tonal contrasts successfully acquired by children. 2 Prior work and motivation Several machine learning models have been trained to classify tonal categories in an unsupervised manner, such as autoencoders [ 5 , 6 , 7 ] , recurrent neural networks [ 8 ] , connectionist temporal classification networks [ 9 ] , wav2vec2.0 [ 10 , 11 , 12 ] , and transformers [ 13 ] as well as convolutional neural networks [ 14 , 15 ] . Although several of these prior models use supervised learning, several unsupervised models have achieved near-100% accuracy on various tonal classification problems [ 16 ] . See Kaur et al. [ 17 ] for an overview of machine learning applied to the automatic recognition and learning of tonal categories, including for Mandarin and other Sinitic languages. Likely due to its status as the most widely spoken native language and thus the most widely spoken tonal language, the plurality of neural network tone classification studies have focused on Mandarin tone [ 5 , 9 , 10 , 11 , 13 , 16 , 18 ] . However, models trained to classify tonal categories, even when unsupervised, are generally not generative : although they learn tonal categories, they cannot generate tonal outputs themselves, and therefore they cannot function as adequate models of human language acquisition, which involves both perception and production. Much research has also focused on achieving successful synthesis of speech audio of tonal languages, which necessarily requires a model that can produce the tonal contrasts of a given language. While speech synthesis of tonal languages must generally implicitly model tone classification to achieve realistic output, relatively few papers focus specifically on assessing how accurately speech synthesis models reproduce the tonal contrasts of the language on which they were trained, with Zhu (2020) [ 19 ] as a notable exception. However, speech synthesis models typically rely either on orthographic or phonemic representations to learn. The ciwGAN models assessed here do not rely on such representations, and thus model the task of acquiring tonal contrasts more realistically than other models that produce synthetic speech. 2.1 Why ciwGAN? To implicitly learn phonological categories while also producing them, we use the categorical InfoWaveGAN (ciwGAN) deep neural network architecture [ 4 ] , a subtype of the InfoWaveGAN architecture that uses categorical one-hot vectors as part of its input vector (Figure 1 ). Because several studies similar to this one have been conducted prior using ciwGAN to learn segmental features, it is worth discussing what research purpose tone serves that features such as aspiration [ 20 ] , nasality [ 21 ] , or vowel harmony [ 22 ] do not. In terms of acoustic identification, tone is one of the simplest contrastive features, being largely distinguishable by F0 trajectory. At the same time, tonal alternations are the most complex processes known in phonology from the standpoint of formal language theory [ 3 , 23 ] . Although this paper focuses on the relatively simple task of learning lexical tone rather than examining these relatively complex tonal alternations such as shifting or spreading, this acts as a baseline for ciwGAN for later research involving more complex tonal alternations. Figure 1: The structure of the ciwGAN [ 4 ] used in training. We leverage the unique benefits of GANs for cognitive modeling enumerated in prior work [ 4 , 20 ] . The similarities to human learning stem from the information to which the network has access and its ability to produce generative output. The model’s learning is completely unsupervised and the data are unlabeled, as is the case for human language learning. Also as in language acquisition, the model must learn the relevant phoneme categories from audio input and then produce the relevant categories as audio output. Most prior research on the machine learning of tone focuses either on classifying tonal categories without being able to produce novel productions of those categories, or the production of tonal categories from labeled data. Autoencoder neural networks augmented with clustering algorithms have achieved unsupervised learning of tonal categories similar to this research [ 6 ] . However, Beguš [ 20 ] notes that autoencoders are not generative in a linguistic sense, as they simply take existing audio as input and transform it. While autoencoders can implicitly learn the acoustic correlates

Read the complete text