Beyond Deep Learning: Speech Segmentation and Phone Classification with Neural Assemblies
Trevor, Vidhyasaharan, Ting · 2026
Read Beyond Deep Learning: Speech Segmentation and Phone Classification with Neural Assemblies by Trevor, Vidhyasaharan, Ting (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16923.txt.
Introduction
Adelson, Sethu, and Dang build a speech system on Assembly Calculus instead of deep learning. Deep learning needs huge datasets and global weight updates, and its features are hard to read. Assembly Calculus uses sparse neuron assemblies, Hebbian plasticity, and winner take all competition. Their framework turns speech into spike patterns and sorts assemblies across areas and timescales. It detects phone boundaries with an F1 score of 0.69 and word boundaries at 0.61, with no weight training. Phone recognition reached 47.5 percent accuracy in their tests.
Catalog record
| CREATOR | Trevor, Vidhyasaharan, Ting |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43956 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16923
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Deep Learning: Speech Segmentation and Phone Classification with Neural Assemblies Trevor, Vidhyasaharan, Ting Abstract Deep learning dominates speech processing but relies on massive datasets, backpropagation, and produces dense representations. Assembly Calculus (AC), which models sparse neuronal assemblies with Hebbian plasticity and winner-take-all competition, offers a biologically grounded alternative. We introduce an AC-based speech processing framework that operates directly on continuous speech by combining three key contributions: (i) neural encoding that converts speech into assembly-compatible spike patterns; (ii) a multi-area architecture organising assemblies across hierarchical timescales and classes; and (iii) cross-area update schemes. Applied to two tasks of boundary detection and segment classification, our framework detects phone (F1=0.69) and word (F1=0.61) boundaries without weight training and achieves 47.5% and 45.1% accuracy on phone and command recognition. These results show that AC-based dynamical systems are a promising alternative to deep learning for speech. Adelson Sethu Dang keywords assembly calculus, speech processing, boundary detection, classification, dynamical systems † † address: 1 The University of Melbourne, Australia 2 The University of New South Wales, Australia † † email: [email protected], [email protected], [email protected] 1 Introduction Deep learning (DL) has significantly transformed speech processing over the last two decades with architectures based on convolutional and recurrent neural networks, and more recently Transformer and Conformer models, now being almost universally adopted for tasks such as automatic speech recognition, speaker verification, and paralinguistic analysis. Large-scale self-supervised approaches such as wav2vec 2.0 [ 3 ] and HuBERT [ 13 ] demonstrate that representations learned from vast amounts of unlabelled audio can be fine-tuned to achieve near–human-level performance on standard benchmarks. However, despite these advances and the models exhibiting high accuracies at a number of benchmarks, when compared to the adaptive, efficient, and structured processing exhibited by the human brain, current deep learning systems reveal several important limitations. Chief amongst them is the extremely resource intensive nature of model training . For instance, a state-of-the-art ASR such as Whisper is trained on 680,000 hours of speech data (approximately 77 years of speech) [ 21 ] , which is orders of magnitude more data than humans need to learn to speak. This limitation in turn is a consequence of other architectural and computational limitations. For example, DL models are trained using stochastic gradient descent which requires all model parameters to be updated during training. Additionally, the internal representations within DL models are typically not sparse, nor do they lend themselves in any obvious manner to composability of the concepts that are being represented such as phonetic content, prosody, or speaker identity. Finally, DL models are predominantly feed-forward models (with the exception of some limited recurrent structures such as RNNs, LSTMs, etc.), which in turn makes them highly suited for running on GPUs but less ideal for problems that are inherently recurrent such as engaging in a speech conversation with no information about when the interaction might end. Furthermore, adding new phones, acoustic categories, or speaker identities usually requires updating shared parameters, often risking catastrophic forgetting [ 14 ] . This tight coupling between representations complicates continual learning and makes it difficult to manipulate or recombine learned phonetic and prosodic primitives without retraining large parts of the system. In contrast, the human brain learns continuously from limited exposure, without making a distinction between “training” and “inference,” and under strict metabolic power constraints [ 15 ] [ 5 ] . Biological learning is inherently incremental and energy-efficient, whereas deep learning pipelines are largely static and resource-intensive. This gap between current deep learning systems and biological intelligence in terms of efficiency, locality, compositionality, and continual adaptation motivates the exploration of alternative computational frameworks inspired more directly by neural principles such as Assembly Calculus (AC), introduced by Papadimitriou et al [ 18 ] . AC provides a computational model of how highly recurrent networks of neurons can represent and manipulate information. In this framework, the basic units are sparse groups of neurons, called assemblies, of which only a few are active at any given time. These assemblies form and change according to simple, local learning rules: when two neurons fire together, the synapse between them is strengthened (Hebbian plasticity), and only the most strongly driven neurons in an area are allowed to fire (winner-take-all inhibition). AC has been applied to language parsing [ 17 , 23 ] , classification [ 6 ] , planning [ 9 ] , and finite state machine simulation [ 7 ] . Because AC employs Hebbian learning with winner-take-all inhibition within local areas, it makes no distinction between training and inference and also sidesteps the requirement to propagate error signals throughout the model. However, AC has important limitations that must be addressed before it can be applied to speech processing. First, all existing AC work assumes discrete, linearly separable input symbols [ 18 ] , whereas real speech signals are continuous and highly coarticulated in time. It is therefore unclear how to construct a binary neural encoding that maps continuous speech features into assemblies while preserving the temporal and spectral structure needed for phone classification. Second, standard AC does not specify how information should be organised across different temporal and spectral scales. Speech processing requires representations that simultaneously capture “when” (e.g., phone or word boundary detection) and “what” (category recognition). It remains unclear whether a single assembly-based representation can support both functions, or how coding schemes should differ across hierarchical linguistic levels. Third, AC relies on local plasticity and does not include an explicit global loss function. As a result, there is no direct mechanism for optimising task-level objectives such as boundary F1 or phone classification accuracy. Existing demonstrations are typically limited to small, idealised symbolic sequences with pre-defined assemblies, and do not address how to organise and stabilise many competing assemblies corresponding to large phone or word inventories, nor do they define how class decisions should be read out from evolving assembly trajectories. In this work, we address these limitations which in turn lets us instantiate AC in concrete, task-driven speech processing systems, targeting two core tasks: temporal segmentation , i.e., identifying the onset times of phones and words in continuous speech without supervised boundary labels; and segment classification , i.e., assigning a phonetic or word-level category label to a given speech segment. We first introduce a binary neural encoding based on probabilistic mel-spectrogram binarisation and population-coded MFCC representations, enabling continuous speech features to be mapped into assembly-compatible inputs. We then develop refractory assembly hierarchies specialised for boundary detection, allowing assemblies to encode temporal segmentation. For classification, we introduce the idea of per-class recurrent areas and a trajectory-based resonance scoring mechanism to read out phone and word identities from assembly dynamics. Together, these components provide a principled implementation of AC for continuous speech, opening up a new line of inquiry that moves beyond conventional deep learning approaches towards more biologically inspired models of speech processing. 2 Assembly Calculus 2.1 Framework In AC, the ‘brain’ is modelled as a collection of areas , each containing n excitatory neurons of which at most k\ll n may fire simultaneously. Areas are connected by sparse random bipartite graphs; we use a fixed in-degree k_{\mathrm{in}} as a deterministic equivalent of the Erdős–Rényi model in the sparse limit. At each discrete time step, neuron v