Enabling Robust In-Context Memory and Rapid Task Adaptation in Transformers with Hebbian and Gradient-Based Plasticity

Siddharth Chaudhary [email protected] · 2025

Read Enabling Robust In-Context Memory and Rapid Task Adaptation in Transformers with Hebbian and Gradient-Based Plasticity by Siddharth Chaudhary [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21908.txt.

Introduction

Siddharth Chaudhary asks whether Transformers learn faster when their connections can change mid-sequence. Brains adapt that way, through synaptic plasticity. Large language models show in-context learning, but their weights stay frozen during use. Living systems keep adapting instead.

Chaudhary adds fast-changing weight modules to decoder-only Transformers. One update rule is Hebbian. It strengthens links between units that fire together, under a modulating signal. The other rule is gradient-based and comes from earlier work. Across copying, regression, and few-shot image tasks, the Hebbian version did best. It reached lower loss and generalized better from few examples. Gradient updates won on long-range credit assignment. Plain static weights sufficed when links were short and simple. That marks where plasticity actually pays.

Catalog record

CREATORSiddharth Chaudhary [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21908

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enabling Robust In-Context Memory and Rapid Task Adaptation in Transformers with Hebbian and Gradient-Based Plasticity Siddharth Chaudhary [email protected] Abstract Large language models display in-context learning as an emergent effect of scale, but they rely on static weights during inference. In contrast, biological systems continually adapt via synaptic plasticity. We investigate whether explicit, biologically inspired plasticity can endow Transformers with faster in-sequence adaptation. To this end, we augment decoder-only Transformers with fast-weight modules updated either by (i) a neuromodulated Hebbian rule or (ii) the gradient-based plasticity mechanism of Duan et al. (2023). Across copying, regression, and few-shot classification tasks (CIFAR-FS, Omniglot), Hebbian plasticity consistently achieves lower loss and stronger few-shot generalization, while gradient-based updates perform best on long-horizon credit assignment. When associations are short and linearly separable, static weights suffice, defining a clear boundary condition for when plasticity helps. Analysis of learned modulatory signals reveals that gradient-based rules maintain large, persistent updates, whereas Hebbian plasticity is sharply gated around salient events. Together, these results show that explicit plasticity complements attention by enabling rapid, task-specific adaptation, and clarify when different plasticity mechanisms are most effective. 1 Introduction Transformers exhibit striking in-context learning (ICL) capabilities: given a few input–output examples, they can perform novel tasks without weight updates. Yet this ability is largely an emergent consequence of scale and training diversity rather than an explicit mechanism for adaptation ( Brown et al. 2020 ; Dai et al. 2023 ) . As a result, the model’s capacity to incorporate new evidence is limited to transient activations within self-attention, with no dedicated process for storing or consolidating information during inference. Biological neural systems, in contrast, continuously modify synaptic strengths in response to experience ( Magee & Grienberger 2020 ) . Such synaptic plasticity enables rapid, context-dependent learning and flexible generalization ( Bittner et al. 2017 ) . Inspired by this principle, we ask: Can explicit plasticity mechanisms improve a Transformer’s ability to adapt within a sequence? To answer this, we augment decoder-only Transformers with fast-weight components that are updated online at each generation step. These updates follow one of two biologically motivated rules: (i) a neuromodulated Hebbian rule , which adjusts weights locally based on correlated pre- and post-synaptic activity, and (ii) a gradient-based plasticity rule ( Duan et al. 2023 ) , which performs local gradient descent on an internally generated target signal. This formulation allows Transformers to adapt during inference, not just across training steps. We evaluate these plastic Transformers on the same family of memory and few-shot learning benchmarks previously used for plastic RNNs, enabling a direct comparison across architectural priors. Our analysis reveals complementary regimes of effectiveness: gradient-based plasticity excels on tasks requiring long-range credit assignment, while Hebbian updates dominate in structured, sparse-supervision settings such as few-shot classification and regression. Our main contributions are: 1. A general framework for incorporating step-wise synaptic plasticity into autoregressive Transformers. 2. A systematic comparison of Hebbian and gradient-based plasticity across copying, regression, and meta-learning benchmarks. 3. Mechanistic insights into how explicit plasticity interacts with self-attention dynamics, clarifying when and why it benefits in-context learning. Together, these results bridge the gap between emergent and mechanistic adaptation, showing that explicit synaptic plasticity can endow Transformers with more efficient, interpretable forms of in-sequence learning. 2 Related Work Our research is situated at the intersection of three active fields: meta-learning, the study of dynamic parameters in artificial networks, and the analysis of in-context learning in Transformers. 2.1 Meta-Learning for Rapid Adaptation Our work falls under the "learning to learn" or meta-learning paradigm Schmidhuber et al. 1997 ; Bengio et al. 1992 , which aims to build models that can adapt to new tasks from limited experience. This field is characterized by a "bi-level" optimization process: an "inner loop" where the model adapts to a specific task, and an "outer loop" where the meta-learner is updated to improve this adaptation process. Several distinct approaches have emerged: • Gradient-based Meta-Learning: This line of work, exemplified by Model-Agnostic Meta-Learning (MAML) Finn et al. 2017 , meta-learns a parameter initialization \theta that can be effectively fine-tuned to a new task with only a few gradient steps. This approach is powerful but typically assumes that the inner-loop task provides an explicit, differentiable loss function. • Metric-based Meta-Learning: Methods like Prototypical Networks Snell et al. 2017 learn a deep embedding space where new examples can be classified based on their proximity to "prototypes" (e.g., class means) derived from the support set. While effective, these methods are often specialized for few-shot classification. • Memory-Augmented Networks (MANNs): These models, such as those proposed by Santoro et al. Santoro et al. 2016 , utilize an external memory (e.g., a Neural Turing Machine’s memory bank) to store task-specific information, which the controller network can read from and write to. • Meta-Learning a Learning Rule: Our work, following Duan et al. Duan et al. 2023 , belongs to a category that meta-learns the learning rule itself . Instead of learning an initialization (like MAML) or an embedding (like ProtoNets), the outer loop meta-trains the parameters of a dynamic update rule (e.g., the connection-specific learning rates \alpha ). The primary advantage of this last approach is its generality and biological plausibility. The plasticity rules are task-agnostic and operate in an unsupervised fashion, driven only by network activity or a self-generated target. This bypasses the need for an explicit, human-defined loss function in the inner loop, allowing the model to adapt to arbitrary sequential experiences. 2.2 Dynamic Parameters and Synaptic Plasticity The core mechanism we explore is the use of "fast weights"—parameters that change on a rapid timescale (i.e., step-by-step) within a single forward pass, as opposed to the "slow weights" updated by the outer loop’s gradient descent. This concept has two main branches in the literature: • Biologically-Inspired Plasticity: This approach draws direct inspiration from neuroscience. Hebb’s rule ("cells that fire together, wire together") is a classic model of such plasticity. In ANNs, this has been adapted to create associative memories Limbacher & Legenstein 2020 ; Schlag et al. 2021b and to meta-learn policies in simple reinforcement learning tasks Najarro & Risi 2020 . The most direct precursors to our work are Miconi et al. Miconi et al. 2018 ; Miconi et al. 2019 and the original paper by Duan et al. Duan et al. 2023 , which applied differentiable, generalized Hebbian rules to RNNs to solve memory and meta-learning problems. • Hypernetworks and Fast Weight Programmers: An alternative, engineering-driven approach is to use a secondary network, or "hypernetwork," to generate the weights of a primary network Ha et al. 2017 . This concept has been adapted for Transformers, notably in "Fast Weight Programmers" Schlag et al. 2021a ; Irie et al. 2021 , where a recurrent network or attention mechanism generates the fast weights used by another component. Our work differs from this latter branch by not using a separate network to generate weights. Instead, we implement specific, local (Hebbian) and global (gradient-based) update rules that modify the weights based on the network’s own activity, as proposed in Duan et al. 2023 . 2.3 Explicit vs. Implicit In-Context Learning in Transformers A significant challenge and inspiration for this work is the remarkable in-context learning (ICL) ability of large-scale Transformers Brown

Read the complete text