Adaptive Hebbian Memory Routing in Vision Transformers for Few-Shot Learning

Mohammed Yusuf Mujawar, Noorbakhsh Amiri Golilarz · 2026

Read Adaptive Hebbian Memory Routing in Vision Transformers for Few-Shot Learning by Mohammed Yusuf Mujawar, Noorbakhsh Amiri Golilarz (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24756.txt.

Introduction

Mohammed Yusuf Mujawar and Noorbakhsh Amiri Golilarz improve few shot image recognition with vision transformers. Few shot models must learn new classes from only a handful of labelled examples. Hebbian fast weight memory can store temporary associations during such an episode. Fixed memory behaviour, though, may suit some tasks poorly. Their method adds a small router network that reads support set features. The router then controls how strongly the Hebbian memory acts.

Three variants are studied: adaptive placement, adaptive plasticity, and fully adaptive routing. Tests cover three backbones under five way one shot evaluation, on Omniglot and CIFAR-FS, plus transfer between them. On the Swin backbone, adaptive plasticity lifts accuracy from about 96.7 to about 96.9 percent. Fully adaptive routing does best, near 96.94 percent, and also cuts inference time. On CIFAR-FS, adaptive variants improve all three backbones. Gains persist as more support examples are added, Mujawar and Golilarz report.

Catalog record

CREATORMohammed Yusuf Mujawar, Noorbakhsh Amiri Golilarz
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24756

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Adaptive Hebbian Memory Routing in Vision Transformers for Few-Shot Learning Mohammed Yusuf Mujawar, Noorbakhsh Amiri Golilarz Abstract Few-shot image recognition requires models to adapt to new classes from a small labeled support set. Hebbian fast-weight memory can provide temporary associative information during an episode, but fixed memory behavior may not be appropriate for every few-shot task. In this work, we propose Adaptive Hebbian Routing for few-shot Vision Transformers. The method uses a lightweight MLP router to control the contribution of Hebbian memory, the strength of memory updates, and the retention of previous memory from support-set features. We study Adaptive Placement, Adaptive Plasticity, and Fully Adaptive Hebbian Routing. Experiments use ViT-Small, DeiT-Small, and Swin-Tiny under 5-way 1-shot evaluation on Omniglot, CIFAR-FS, and cross-domain transfer from CIFAR-FS to Omniglot. In the direct Swin comparison, fixed and adaptive Hebbian variants use the same memory location. Adaptive Plasticity improves the fixed Hebbian result from 96.74% to 96.92%, while Fully Adaptive Routing achieves the best result at 96.94%. The fully adaptive Swin model also reduces inference time from 16.51 ms to 14.05 ms relative to fixed Hebbian Swin. On CIFAR-FS, adaptive variants improve performance across all three backbones, and the multi-shot evaluation shows that these gains remain useful as the number of support examples increases. These results show that adaptive plasticity and adaptive memory activation can improve few-shot Transformer representations beyond fixed Hebbian behavior. I Introduction Few-shot image recognition requires a model to recognize new classes from only a small labeled support set. This setting is difficult because standard deep networks store most of their knowledge in slow weights that remain fixed during inference. Metric-based methods reduce this difficulty by comparing query examples with the small support set. For example, Prototypical Networks classify a query according to its distance from class prototypes formed from support embeddings [ 17 ] . However, the success of this process still depends on whether the backbone can form useful task-specific representations from the current episode. Vision Transformers provide strong image representations, but they are mostly static during inference. ViT represents images as sequences of patch tokens [ 4 ] , DeiT improves data-efficient Transformer training [ 19 ] , and Swin Transformer uses a hierarchical shifted-window design for visual recognition [ 9 ] . These architectures are effective feature extractors, but few-shot learning requires them to adapt to a new support set for every episode. This motivates mechanisms that can store temporary task-specific information without updating the full backbone. Hebbian fast weights provide one way to add this temporary memory. Fast weights can store recent information on a shorter time scale than standard model parameters [ 8 , 1 ] , while Hebbian learning forms associations between co-activated representations [ 7 ] . This idea has been used for meta-learning and one-shot binding [ 12 ] , and fast-weight formulations have also been connected to attention mechanisms [ 15 ] . These studies suggest that associative memory can help a model adapt within an episode by storing information from the support set and using it when processing queries. However, the usefulness of Hebbian memory depends on how it is added to the backbone. A recent study of Hebbian Fast-Weight modules in few-shot Vision Transformers showed that fixed memory placement can produce very different effects across architectures [ 11 ] . In particular, fixed Hebbian memory was effective for Swin, while dense per-block Hebbian insertion degraded ViT and DeiT. This result suggests that selecting a useful memory location is important, but it also raises a second question: once a location is selected, should the Hebbian pathway use the same activation, update strength, and memory retention for every few-shot episode? In this work, we propose Adaptive Hebbian Routing , a lightweight approach that makes the Hebbian memory pathway episode-conditioned. As illustrated in Fig. 1 , an adaptive MLP router receives support-set features from a selected backbone location and predicts three control values. The first controls how strongly the Hebbian memory output contributes to the current representation. The second controls the plasticity rate used to write new support-set associations. The third controls memory retention during the episode. This design allows the model to change how it uses temporary associative memory according to the current support set, rather than applying fixed Hebbian behavior to every task. The main contributions of this paper are as follows: • We propose Adaptive Hebbian Routing, which uses a lightweight MLP router to control Hebbian memory activation, plasticity strength, and memory retention from support-set features. • We study Adaptive Placement, Adaptive Plasticity, and Fully Adaptive Hebbian Routing to separate the effect of memory activation from episode-specific memory updates. • We provide a direct Swin comparison in which fixed and adaptive Hebbian variants use the same memory location, isolating the effect of adaptive control. • We show that Adaptive Plasticity improves fixed Hebbian Swin, while Fully Adaptive Hebbian Routing achieves the strongest Swin accuracy and improves inference speed. • We evaluate accuracy, parameter count, and inference time across Omniglot, CIFAR-FS, and cross-domain transfer from CIFAR-FS to Omniglot. The remainder of this paper is organized as follows. Section II reviews related work on few-shot learning, Hebbian fast weights, and adaptive routing. Section III presents the proposed Adaptive Hebbian Routing method and describes its integration into Vision Transformer backbones. Section IV describes the datasets, evaluation protocol, model variants, and implementation details. Section V reports the experimental results and analysis. Section VI discusses the main findings and limitations, and Section VII concludes the paper. II Related Work II-A Few-Shot Learning with Learned Representations Few-shot learning aims to classify novel categories from a small labeled support set. Early metric-based methods such as Matching Networks learn to compare a query example with labeled support examples directly [ 20 ] . Prototypical Networks simplify this setting by representing each class with a prototype computed from its support embeddings and assigning queries according to distance from those prototypes [ 17 ] . Relation Networks instead learn a comparison function between support and query representations [ 18 ] . Other approaches focus on rapid parameter adaptation, including Model-Agnostic Meta-Learning (MAML) [ 6 ] , or on adapting the embedding space to each task using set-to-set functions [ 22 ] . Our work follows the episodic prototype-based setting, but focuses on improving the backbone representation through temporary adaptive memory rather than changing the classifier itself. II-B Fast Weights and Hebbian Plasticity Fast-weight models separate long-term knowledge stored in slow parameters from temporary associations formed during processing. Early work described fast weights as a rapidly changing memory overlay that can support temporary learning and binding [ 8 ] . Later work used fast weights to attend to recent information [ 1 ] and showed that Hebbian fast weights can bind representations and labels within a meta-learning task [ 12 ] . The connection between attention and fast-weight programming was further developed by work showing that key-value outer-product updates can be interpreted as a fast associative memory [ 15 ] . Differentiable plasticity also demonstrated that plastic connections can be optimized jointly with neural network weights through gradient-based training [ 10 ] . The fixed Hebbian Transformer baseline most closely related to our work applied Hebbian Fast-Weight modules to Vision

Read the complete text