Hebbian learning the local structure of language

P. Myles Eugenio · 2025

Read Hebbian learning the local structure of language by P. Myles Eugenio (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-02057.txt.

Introduction

Brain learning is local and unguided, yet it yields language, and Myles Eugenio derives a language model built on those same limits, in work from twenty twenty-five. One stack of cells learns to chop text streams into word parts. A second set of cells binds those raw parts into a meaning-bearing token.

The model learns on the fly without forgetting, and its tokens always split back into a base alphabet. It can even blend traits from several tongues. Most striking, its word-forming patterns match real languages with no language data fed in. Eugenio offers the model as a root account of why speech falls into words at all.

Catalog record

CREATORP. Myles Eugenio
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-02057

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hebbian learning the local structure of language P. Myles Eugenio Abstract Learning in the brain is local and unsupervised (Hebbian). We derive the foundations of an effective human language model inspired by these microscopic constraints. It has two parts: (1) a hierarchy of neurons which learns to tokenize words from text (whichiswhatyoudowhenyoureadthis); and (2) additional neurons which bind the learned symanticless patterns of the tokenizer into a symanticful token (an embedding). The model permits continuous parallel learning without forgetting; and is a powerful tokenizer which performs renormalization group. This allows it to exploit redundancy, such that it generates tokens which are always decomposable into a basis set (e.g an alphabet), and can mix features learned from multiple languages. We find that the structure of this model allows it to learn a natural language morphology without data. The language data generated by this model predicts the correct distribution of word-forming patterns observed in real languages, and further demonstrates why microscopically human speech is broken up into words. This model provides the basis for understanding the microscopic origins of language and human creativity. I Introduction In the late 1970’s, deaf Nicaraguan school children invented an indigenous sign language unrelated to any already existing language (spoken or signed) Kegl and Iwata 1989 ; Senghas 1995 ; Senghas and Coppola 2001 . These children had no previous exposure to a developed language. The earliest stages of this language developed spontaneously out of their collective interactions; later becoming increasingly nuanced and systematized as younger incoming generations acquired it from older students Senghas and Coppola 2001 . The formation of Nicaraguan Sign Language (NSL) is perhaps the sharpest example of human learning absent data. Its circumstances are unique to a disability, making its emergence unbiased by existing data. Similar learning of novel language has occurred throughout history, such as in the formation of creoles Kouwenberg and Singler 2018 ; Mcwhorter and Good 2012 . This process of rapid language generation & change is typically driven by the youngest learners – a surprising contrast with data-hungry large language models (LLMs). For contrast, LLMs struggle to learn existing creoles due to the limitations on available data Lent et al. 2021 ; Lent et al. 2024 ; Feldman and Zhang 2020 . Though modern generative AI have improved impressively in recent years Wei et al. 2023 ; Long 2023 ; Grattafiori et al. 2024 ; DeepSeek-AI et al. 2025 , the predicted rate of improvement is power-law in the data and compute Kaplan et al. 2020 ; Vyas et al. 2023 ; Hestness et al. 2017 ; Bahri et al. 2024 ; Aghajanyan et al. 2023 . Training such models is only possible because of the mass recording and centralization of human data, which are curiously not the conditions which led to that data. This opens a broader question into the existence and origin of language and its data. NSL did not exist in the ’60s but existed by the ’70s. In other words: How do we go from a universe without language to a universe with language? Dense models which require data in order to produce language cannot account for its existence. From this light, it becomes apparent that NSL and the creoles are the tip of the iceberg: We speak all these languages. Where did all the data come from? In this work, we provide the essential framework for answering this question. To do this, we reexamine the microscopic conditions of human speech. Here microscopic meaning in the most detailed sense at the 2-neuron level. At this level, the only justified mechanism for learning is Hebbian learning, which describes the tendency for neurons to correlate their firing. In the language of the Hopfield networks Hopfield 1982 ; Krotov and Hopfield 2021 , these correlations between the firing state vectors u_{j} (with firing rate v_{j}\equiv v_{j}(u_{j}) ) of different neurons are encoded in a matrix g_{jk} , which evolves as \displaystyle\tau_{g}\dot{g}_{jk}+g_{jk}=v_{j}(u_{j})v_{k}(u_{k}) (1) over timescale \tau_{g} . The right-hand side of Eqn 1 can be understood as arising from the gradient descent of a network energy Krotov and Hopfield 2021 , H=\sum_{jk}v_{j}g_{jk}v_{k} , which encodes the local (2-point) interaction between neurons. Learning in this paradigm is local and unsupervised . Unlike the microscopic scale, the correlations between the tokens of speech are longer than 2. For example, the 3-point correlation str in English strength . (The tokens here being the alphabetic letters.) Even longer correlations make up the morphology of strong , strongest , & strength . More generally, speech is composed of correlated strings of arbitrary length ( N -point). A coherent sentence requires that its letters be correlated with the other letters of their word, as well as with the letters which compose the words elsewhere in the sentence. Long correlations are a defining property of a successful language model, without which it would exhibit a lack of coherence and an inability to stay on topic. This makes understanding why such correlations arise, in spite of the microscopic constraints, a central puzzle in understanding human speech. Large language models are aided in this task by having attention mechanisms with non-linear activation functions, like softmax, which generate non-linear superpositions during inference Vaswani et al. 2017 . The transformer architecture has already been reformulated as a modern Hopfield network Ramsauer et al. 2021 ; Hoover et al. 2023 , and it has been argued that such models are biologically plausible because they can be derived from local theories under some assumptions Krotov and Hopfield 2021 ; Kozachkov et al. 2023 ; Rahimi and Recht 2007 ; Salinas and Abbott 1994 . This includes allowing for exponential memory capacity for dense autoassociative memories Ramsauer et al. 2021 ; Krotov and Hopfield 2021 . However, while these black boxes can successfully learn long correlations, they make no predictions, and do not teach us anything about the microscopic origins of language and its structure. If human language memory was exponential, then a language with only 10 syllables could communicate 10^{5} messages in a string of length 5 . Instead human speech is broken up into discrete locally-correlated chunks (words), whose entropy for a length N string is less than exponential HERDAN 1958 ; WILLIAMS 1940 ; Limpert et al. 2001 ; Newman 2006 ; Eugenio 2023 . We then ask if it is possible to learn correlated strings without violating the unsupervised and local constraints of the biology. We find that it is possible with the aid of a hierarchy of local interactions, each of the form Eqn 1 . The resulting model learns words by strengthening correlations between neighboring tokens in the text, starting first with letter-letter correlations (bigrams). These learned bigrams are then used to define compound tokens, which are used to train correlations ( n -grams) at the next level of the hierarchy. This process is repeated, generating a series of projector maps used for tokenization. Such a model is an n -gram model Che et al. 1953 which learns by playing a game akin to byte pair encoding (BPE) Gage 1994 ; except with the additional constraint that all learned n -grams be composed only of learned n-1 grams, which are the nodes of a directed acyclic graph.

Read the complete text