Meta-Learning Neural Mechanisms rather than Bayesian Priors

Michael Eric Goodale, Salvador Mascarenhas, Yair Lakretz · 2025

Read Meta-Learning Neural Mechanisms rather than Bayesian Priors by Michael Eric Goodale, Salvador Mascarenhas, Yair Lakretz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16048.txt.

Introduction

Michael Goodale and co-authors ask what meta-learning actually installs in a neural network, and find machinery, not taste. Meta-learning was sold as giving networks human-like simplicity biases, the way children generalize language from little data. Training networks on formal languages arranged around simplicity, the team tested that story.

The models were not learning simplicity priors at all. Instead, meta-training imprinted concrete neural mechanisms, counters and similar devices, that act like cognitive primitives on later tasks. More startling: meta-training on one well-chosen formal language helped as much as training on five thousand, provided that one language forced the useful mechanisms to form. The authors draw practical lessons for cheaper meta-learning and a theoretical bridge between symbolic accounts of mind and neural hardware.

Catalog record

CREATORMichael Eric Goodale, Salvador Mascarenhas, Yair Lakretz
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16048

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Meta-Learning Neural Mechanisms rather than Bayesian Priors Michael Eric Goodale, Salvador Mascarenhas, Yair Lakretz Abstract Children acquire language despite being exposed to several orders of magnitude less data than large language models require. Meta-learning has been proposed as a way to integrate human-like learning biases into neural-network architectures, combining both the structured generalizations of symbolic models with the scalability of neural-network models. But what does meta-learning exactly imbue the model with? We investigate the meta-learning of formal languages and find that, contrary to previous claims, meta-trained models are not learning simplicity-based priors when meta-trained on datasets organised around simplicity. Rather, we find evidence that meta-training imprints neural mechanisms (such as counters) into the model, which function like cognitive primitives for the network on downstream tasks. Most surprisingly, we find that meta-training on a single formal language can provide as much improvement to a model as meta-training on 5000 different formal languages, provided that the formal language incentivizes the learning of useful neural mechanisms. Taken together, our findings provide practical implications for efficient meta-learning paradigms and new theoretical insights into linking symbolic theories and neural mechanisms. 1 Introduction In the past decade, enormous advances in neural network models have led to a novel neural-network orthodoxy perhaps best exemplified by Richard Sutton’s Bitter Lesson ( Sutton 2019 ) . The now prevailing view has been to eschew purpose-built architectures in favour of scale in larger models and larger training datasets rather than narrow purpose-built architectures. Nevertheless, humans’ capacity to generalize from very few examples has remained an elusive and intriguing goal for both AI practitioners and cognitive scientists sympathetic to connectionism. Yang and Piantadosi 2022 introduced a symbolic model to capture aspects of humans’ basic generalisations from small data on simple formal languages. They use a “language of thought” model which combines simple cognitive “primitives” (e.g. simple functions like logical AND, getting the first element of a list, and so on) into symbolic programs explored via Bayesian-inference techniques. The model can then “learn”, given some data, by finding a program which generates the underlying data, potentially generalizing correctly into out-of-distribution data. Crucially, the model has a Bayesian prior for simpler (e.g. shorter) programs. The model can only learn effectively with toy formal grammars, for example it perfectly generalizes from the minuscule dataset, \{a,aa,aaa\} , to the formal grammar a^{n} . Despite the model’s performance on simple grammars, its symbolic nature makes it difficult to scale up to more complicated material such as natural language, because the non-differentiable search over programs is intractable and cannot exploit gradient-descent. One potential way to square the circle of symbolic modeling’s theoretically-motivated generalizations with neural-network approaches’ scaleable learning is meta-learning. Meta-learning is an increasingly popular approach among cognitively-minded researchers to address humans’ capacity for systematic generalization while staying within the familiar realm of mainstream neural-network techniques ( Lake and Baroni 2023 ; Binz et al. 2024 ; McCoy and Griffiths 2023 ; McCoy et al. 2020 ) . This approach has models learn baseline expectations, in the form of “Bayesian priors” or “inductive biases,” approximating and hopefully reproducing the language-readiness of the human child’s brain. Meta-learning approaches thus avoid the challenges of selecting and hard coding an appropriate a priori architecture, while leveraging the remarkable learning abilities of neural networks. Consequently, several cognitive scientists have proposed that meta-learning of one form or another may provide solutions to the problems of out-of-distribution generalization on the basis of limited data ( Lake and Baroni 2023 ; Binz et al. 2024 ) . McCoy and Griffiths 2023 use meta-learning in the special case of language learning: they use meta-learning techniques to, as they put it, “distill” a Bayesian prior for a particular kind of simplicity into a neural network. They use model-agnostic meta-learning ( Finn et al. 2017 , MAML;) to meta-train LSTMs to learn simple formal languages from small amounts of data in the same schema as Yang and Piantadosi 2022 . Their model is meta-trained on formal languages sampled with preference for simplicity, and then trains on datasets like those from Yang and Piantadosi 2022 . Their model shows impressive improvements in performance compared to an un-metatrained neural network, with generalizations from small data comparable to Yang and Piantadosi 2022 ’s ( Yang and Piantadosi 2022 ) symbolic model. They describe their model as distilling a Bayesian prior for simplicity into a connectionist model, thus benefiting from the flexibility provided by neural networks. In other words, they claim the model learns to mimic its meta-training distribution when trained on later tasks, thereby following the simplicity prior of its meta-training dataset. Here we examine McCoy and Griffiths 2023 ’s ( McCoy and Griffiths 2023 ) proposal for meta-learning in the case of formal languages in detail, and demonstrate that their conclusion is premature. We provide an alternative mechanistic view , which, in contrast to the simplicity-bias view , suggests that meta-training encourages models to learn useful neural mechanisms (e.g. counters) and that meta-learning does not distill simplicity biases. The neural mechanisms then function as available “neural primitives” during task acquisition. We contrast the mechanistic and simplicity-bias views by testing each of their predictions, as spelled out in Table 1 , and find converging evidence in support of the mechanistic view, but not the simplicity-bias one: First, models can achieve the same behavioral outcomes using meta-learning datasets that in no way follow a Bayesian simplicity prior, in contrast to the prediction of the simplicity-bias view. Crucially, second, we find that a single formal language can serve as a sufficiently rich meta-learning dataset in certain cases. These can be explained by the mechanistic view but not the simplicity-bias view. Finally, manipulating the types of mechanisms a neural architecture can develop (e.g., preventing counting by replacing LSTMs by GRU units), we show that meta-learning effects disappear, as predicted by the mechanistic but not the simplicity-bias view. Simplicity-bias view Mechanistic-complexity view Meta-trained models learn to learn with a Bayesian simplicity prior. Meta-trained models learn useful neural-mechanisms. 1a) Simplicity-focus datasets will help the model learn to make the simplest generalization in later training. 1b) Datasets without a prior for simplicity will make incorrect generalizations and perform worse. 2a) For formal tasks, data-diversity is not particularly important as long as the dataset requires a useful mechanism. 2b) A meta-training dataset is useful only if the neural-architecture is able to acquire the relevant mechanisms. Table 1: Predictions of the simplicity-bias view and the mechanistic view . We find evidence in support of the mechanistic-complexity view (predictions (2a) and (2b)) and against the simplicity-bias view (predictions (1a) and (1b)). 2 Meta-Learning a simplicity prior? The received wisdom today is that the best kinds of generalizations are those that are implied by the Kolmogorov complexity of the data ( Solomonoff 1964 ; Kolmogorov 1963 ; Chater 1999 ; Chater and Vitányi 2003 ) . The idea is roughly that the shortest program which can generate your data will have the right kinds of generalizations. While Kolmogorov complexity itself is non-computable, different approximations of it have been used before for neural-networks such as Minimum Description Length ( Lan et al. 2022 ) . Unfortunately, these approximations tend themselves to be non-differentiable, leading them to be unsuitable for gradient-descent and

Read the complete text