NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptation
Yuxin Yang, Haoran Zhang, Mingxuan Li, Jiachen Xu, Ruoxi Shen, Zhenyu Wang, Tianhao Liu, Siqi Chen, Weilin Huang · 2026
Read NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptation by Yuxin Yang, Haoran Zhang, Mingxuan Li, Jiachen Xu, Ruoxi Shen, Zhenyu Wang, Tianhao Liu, Siqi Chen, Weilin Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12378.txt.
Introduction
Yang and co-authors add context-aware neuromodulation to LoRA for multi-task adaptation. LoRA variants tune large models cheaply, but FlyLoRA routes by static magnitude and ignores input context. NeuroLoRA keeps frozen random projections and adds a light learnable gate inspired by brain neuromodulation. That gate rescales the projection space by context before experts are chosen. A contrastive orthogonality loss pushes expert subspaces apart for cleaner task splits. Tests on MMLU, GSM8K, and ScienceQA beat FlyLoRA and other baselines across single-task, merging, and continual settings.
Catalog record
| CREATOR | Yuxin Yang, Haoran Zhang, Mingxuan Li, Jiachen Xu, Ruoxi Shen, Zhenyu Wang, Tianhao Liu, Siqi Chen, Weilin Huang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32320 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12378
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptation Yuxin Yang, Haoran Zhang, Mingxuan Li, Jiachen Xu, Ruoxi Shen, Zhenyu Wang, Tianhao Liu, Siqi Chen, Weilin Huang Abstract Parameter-Efficient Fine-Tuning (PEFT) techniques, particularly Low-Rank Adaptation (LoRA), have become essential for adapting Large Language Models (LLMs) to downstream tasks. While the recent FlyLoRA framework successfully leverages bio-inspired sparse random projections to mitigate parameter interference, it relies on a static, magnitude-based routing mechanism that is agnostic to input context. In this paper, we propose NeuroLoRA , a novel Mixture-of-Experts (MoE) based LoRA framework inspired by biological neuromodulation—the dynamic regulation of neuronal excitability based on context. NeuroLoRA retains the computational efficiency of frozen random projections while introducing a lightweight, learnable neuromodulation gate that contextually rescales the projection space prior to expert selection. We further propose a Contrastive Orthogonality Loss to explicitly enforce separation between expert subspaces, enhancing both task decoupling and continual learning capacity. Extensive experiments on MMLU, GSM8K, and ScienceQA demonstrate that NeuroLoRA consistently outperforms FlyLoRA and other strong baselines across single-task adaptation, multi-task model merging, and sequential continual learning scenarios, while maintaining comparable parameter efficiency. 1 Introduction The paradigm of pre-training followed by task-specific fine-tuning has become the dominant approach for deploying Large Language Models (LLMs) 3 ; 1 ; 11 . However, Full Fine-Tuning (FFT) of models with billions of parameters incurs prohibitive computational and storage costs, motivating the development of Parameter-Efficient Fine-Tuning (PEFT) methods 13 ; 19 ; 21 . Among these, Low-Rank Adaptation (LoRA) 14 has emerged as the de facto standard, freezing the pre-trained weights and injecting trainable low-rank decomposition matrices into the model’s linear layers. Despite its efficiency, standard LoRA assumes a single global low-rank update shared across all inputs. This design leads to intra-task interference —where diverse input patterns within a single task compete for the same limited rank capacity—and inter-task interference when merging adapters trained on different tasks 40 ; 39 ; 45 . To address these limitations, Mixture-of-Experts (MoE) architectures have been integrated into the LoRA framework 8 ; 38 ; 20 , enabling conditional activation of task-specific parameter subsets. A recent advancement, FlyLoRA 45 , draws inspiration from the fruit fly olfactory circuit 6 to replace trainable routers with a frozen sparse random projection matrix. By selecting experts based on the magnitude of the projected input (a “Winner-Take-All” mechanism analogous to Kenyon Cell activation), FlyLoRA achieves strong task decoupling and enables training-free model merging. However, FlyLoRA’s reliance on a fixed random projection represents a purely structural imitation of biological neural circuits, overlooking a critical functional component: neuromodulation 28 . In biological brains, neural responses are dynamically modulated by neurotransmitters (e.g., dopamine, acetylcholine) that alter neuronal sensitivity based on internal states and external context 9 . FlyLoRA’s routing is reactive to the current token’s magnitude profile but agnostic to the broader semantic context. This static routing can be suboptimal for tasks requiring contextual reasoning, where the appropriate expert specialization depends on the meaning of preceding tokens rather than the current token’s magnitude alone. To bridge this gap, we propose NeuroLoRA (Neuromodulated Low-Rank Adaptation). NeuroLoRA preserves the computational efficiency and merging compatibility of FlyLoRA’s frozen sparse projection while augmenting it with a Context-Aware Neuromodulation Gate . This lightweight gating component dynamically rescales the axes of the random projection space based on input context, effectively modulating the routing decision boundary without altering the projection matrix itself. Furthermore, we introduce a Contrastive Orthogonality Loss ( \mathcal{L}_{orth} ) that explicitly enforces separation between expert subspaces, providing stronger guarantees than the probabilistic near-orthogonality of random projections alone. This design additionally confers natural advantages for continual learning, as the enforced subspace separation mitigates catastrophic forgetting 17 when the model is sequentially adapted to new tasks. Our contributions are summarized as follows: • We identify the limitation of static magnitude-based routing in FlyLoRA and propose a biologically motivated solution grounded in the principle of neuromodulation. • We introduce NeuroLoRA, which incorporates a lightweight context-aware gating mechanism to dynamically modulate the frozen projection space, enabling context-sensitive expert activation while preserving training-free merging compatibility. • We propose a Contrastive Orthogonality Loss that explicitly penalizes expert subspace overlap, enhancing task decoupling for both multi-task merging and continual learning. • Extensive experiments on three benchmarks (MMLU, ScienceQA, GSM8K) demonstrate that NeuroLoRA outperforms FlyLoRA and other strong baselines in single-task, multi-task merging, and continual learning settings. 2 Related Work 2.1 Parameter-Efficient Fine-Tuning PEFT aims to adapt pre-trained LLMs to downstream tasks while updating only a small fraction of parameters. Early approaches include Adapter layers 13 , which insert bottleneck modules between Transformer 35 layers, and Prefix-Tuning 21 , which prepends learnable continuous tokens to the input. Prompt Tuning 19 further simplifies this by learning soft prompts in the input embedding space. 14 introduced LoRA, which injects low-rank matrices B\in\mathbb{R}^{d_{out}\times r} and A\in\mathbb{R}^{r\times d_{in}} into frozen linear layers, where r\ll\min(d_{in},d_{out}) . Subsequent work has focused on improving rank allocation and training dynamics. AdaLoRA 42 dynamically prunes singular values via importance scoring to allocate rank budget across layers. QLoRA 7 combines 4-bit quantization of the base model with LoRA adapters, substantially reducing memory footprint. DoRA 24 decomposes the weight update into magnitude and directional components to improve learning stability. While effective for single-task adaptation, these methods are susceptible to parameter interference in multi-task and continual learning settings 45 ; 37 . 2.2 Mixture-of-Experts for LoRA Mixture-of-Experts (MoE) architectures conditionally activate subsets of parameters based on input characteristics 34 ; 10 ; 18 , enabling increased model capacity without proportional computational cost. Integrating MoE with LoRA has emerged as a principled approach to multi-task adaptation. LoRAMoE 8 employs a localized balancing constraint to prevent expert collapse during instruction tuning. MoLE 38 and MixLoRA 20 utilize top- k gating networks to route tokens to different LoRA expert modules. 41 investigated the parameter efficiency limits of MoE-LoRA configurations. AdaMix 36 randomly mixes adapter experts during training for improved generalization. However, these methods rely on explicit, trainable router networks (typically linear layers followed by softmax), which introduce additional learnable parameters and—more critically—make training-free model merging infeasible since the routing logic is learned and task-specific. NeuroLoRA differs fundamentally by retaining a frozen projection for routing while introducing dynamic modulation. 2.3 Bio-Inspired Representations and Model Merging The fruit fly olfactory circuit has inspired efficient locality-sensitive hashing algorithms 6 . The core principle—projecting inputs into a high-dimensional sparse space to separate overlapping patterns—was applied to LoRA by FlyLoRA 45 , which treats the down-projection matrix A as a frozen sparse random projection and uses magnitude-based Winner-Take-All activation for expert selection. This design inherently supports Model Merging techniques such as Task Arithmetic 15 and TIES-Merging 39 , due to the approximate orthogonality of high-dimensional random vectors. Beyond task decoupling and merging, recent work has revealed that the structural properties of the fly olfactory circuit—in particular, its sparse, high-dimensional, and approximately orthogonal random projections—naturally mitigate the stability-plasticity dilemma central to continual learning 43 . Fly-CL 44 further extended this insight into a practical continual representation learning framework, demonstrating that fly-inspired decorrelation mechanisms can significantly reduce inter-task interference while maintaining training efficiency. Our work identifies a critical gap in FlyLoRA: while it imitates the structure (fixed connectivity) of the fly olfactory circuit, it omits the functional dynamics of biological