Walking the Weight Manifold: a Topological Approach to Conditioning Inspired by Neuromodulation

Ari S. Benjamin, Kyle Daruwalla, Christian Pehle, Anthony Zador · 2025

Read Walking the Weight Manifold: a Topological Approach to Conditioning Inspired by Neuromodulation by Ari S. Benjamin, Kyle Daruwalla, Christian Pehle, Anthony Zador (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22994.txt.

Introduction

Brains switch tasks not by rewiring inputs but by retuning synapses, using chemical modulators like serotonin. Ari Benjamin and co-authors borrow that trick in 2025: instead of training one weight set per task, they train a smooth surface in weight space. The surface is a manifold whose shape, line, circle, or torus, encodes how tasks relate. A formal descent rule moves the whole surface downhill under a volume budget.

At run time, context picks a point on the surface, and those weights run the network. Manifold-trained nets beat input-conditioned twins and generalized better to unseen cases. Learning one task bends the whole surface, spreading the lesson to neighbors.

Catalog record

CREATORAri S. Benjamin, Kyle Daruwalla, Christian Pehle, Anthony Zador
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22994

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Walking the Weight Manifold: a Topological Approach to Conditioning Inspired by Neuromodulation Ari S. Benjamin, Kyle Daruwalla, Christian Pehle, Anthony Zador Abstract One frequently wishes to learn a range of similar tasks as efficiently as possible, re-using knowledge across tasks. In artificial neural networks, this is typically accomplished by conditioning a network upon task context by injecting context as input. Brains have a different strategy: the parameters themselves are modulated as a function of various neuromodulators such as serotonin. Here, we take inspiration from neuromodulation and propose to learn weights which are smoothly parameterized functions of task context variables. Rather than optimize a weight vector, i.e. a single point in weight space, we optimize a smooth manifold in weight space with a predefined topology. To accomplish this, we derive a formal treatment of optimization of manifolds as the minimization of a loss functional subject to a constraint on volumetric movement, analogous to gradient descent. During inference, conditioning selects a single point on this manifold which serves as the effective weight matrix for a particular sub-task. This strategy for conditioning has two main advantages. First, the topology of the manifold (whether a line, circle, or torus) is a convenient lever for inductive biases about the relationship between tasks. Second, learning in one state smoothly affects the entire manifold, encouraging generalization across states. To verify this, we train manifolds with several topologies, including straight lines in weight space (for conditioning on e.g. noise level in input data) and ellipses (for rotated images). Despite their simplicity, these parameterizations outperform conditioning identical networks by input concatenation and better generalize to out-of-distribution samples. These results suggest that modulating weights over low-dimensional manifolds offers a principled and effective alternative to traditional conditioning. 1 Introduction Conditioning is essential in the neural network toolbox. For example, an image generator might be conditioned on which category of image to generate or with which style. A running robot might be conditioned on a desired running speed or its goal location. In each of these examples, the sub-tasks are so closely related—relative to all possible functions—that it is more efficient to use a single neural network, provided that one can ensure that learning in one context appropriately generalizes to learning in other contexts. One way to encourage cross-task generalization would be to take advantage of the clear relationships between these tasks. Here, we pay particular attention to the topology of sub-tasks. Running speed is a 1D line through the task space of ambulating. Navigation on a map, conditioned on location, is a 2D operation over a sheet. Classifying images of 3D objects taken from arbitrary orientations lies on the surface of a sphere in task space. An inductive bias which captures these topologies would ensure that knowledge transfers between tasks while enforcing that the true task topology constrains the possible learned input/output mappings. Figure 1: a. Traditional approaches map all conditions in task space to a single network in weight space. Here, conditioning corresponds to the noise added to an image (i.e. input uncertainty) in a classification task. b. Our approach maps various conditions to a parameterized manifold in weight space with known topology chosen to match the task topology. For the task in a , this corresponds to a line, but for alternative tasks like rotations of an input image, an ellipse is more appropriate. c. Manifolds are optimized via our proposed steepest descent rule such that they minimize the volumetric distance between steps. Here, we illustrate this by projecting an ellipse manifold of weights into principal component space during learning. The weights correspond to the first convolutional layer of a network trained to classify CIFAR-10 images. Here, we realize this strategy by hypothesizing that for each manifold of tasks there exists a manifold of corresponding topology in weight space which implements that range of tasks. For example, a topological sphere in weight space likely implements the set of functions for classifying objects rotated in 3D space. More formally, assuming smoothness over the weight and task manifolds, and a topology over the task manifold, our “manifold hypothesis” postulates a homeomorphism between both manifolds. Fig. 1 a and b illustrate this hypothesis and how it differs from a traditional view on neural network weights and conditioning. To empirically evaluate this hypothesis, we formalize this notion and derive a learning rule for optimizing manifolds. This rule is analogous to gradient descent, whose implicit learning biases are arguably the key to success in high-dimensional optimization. Just as gradient descent effectively minimizes the path length through weight space, our rule pushes a manifold in the direction of steepest descent while minimizing its total volumetric movement through weight space. This ensures that learning pressure at one location of the manifold does not push other locations on the manifold in very far directions. This learning rule is designed to respect Occam’s razor, minimally changing the function across the manifold in response to new data anywhere. While it might not be obvious that lines, circles, or other simple shapes exist in weight space that implement realistic functions, we find empirically that such solutions do indeed exist. Fig. 1 c demonstrates how the rule manipulates the structure of the manifold while taking minimal volumetric steps through weight space (shown here for an image classification network’s first layer weights when trained on CIFAR-10 ( krizhevsky_learning_2009 ) ). Furthermore, we demonstrate a potential benefit of conditioning via modulation: generalization to contexts not seen during training. Just as humans must learn while calm but perform while nervous or angry, the modulation of artificial networks allows them to generalize performance to areas in which the involved task slightly differs. In brief, our major contributions include: • a neuro-inspired formalism for topologically constrained weight manifolds • a corresponding steepest descent rule for updating manifolds in principled manner • computationally tractable instances of our rule for simple manifolds (e.g. a line, ellipse, etc.) • practical implementations of our rule that leverage existing automatic differentiation libraries • experimental evaluations demonstrating when the proposed rule is effective and ineffective; notably, that it generalizes to novel conditions unseen in the training data relative to vanilla gradient descent and traditional conditioning 1.1 Inspiration from neuroscience The proposed abstraction of weight manifolds reflects a wide set of studies on the physiological properties of neurons and small circuits. After it was observed in the early 20th century that neurons integrate information from their synaptic inputs—directly leading to the earliest generations of neural network models ( mcculloch_logical_1943 ) —it later become clear that nearly all key parameters of neural systems are, in fact, functions of various neuromodulators ( harris-warrick_modulation_1991 ) . While this work vastly complicated the modeling of small biological circuits, it revealed a general biological capability for reusing neural circuits for different purposes in different behavioral states ( marder_neuromodulation_2012 ) . The mechanistic effects of neuromodulation are profound and diverse. The excitability of neurons are affected, as well as specific synapses and specific ion channels ( klein_serotonin_1982 ; levitan_modulation_1988 ) . Effects vary depending upon the cell type in question, sometimes in opposite directions for the same neuromodulator ( gerfen_modulation_2011 ) . Furthermore, the delivery of neuromodulation can be extremely targeted to single neurons (i.e. cotransmission) ( nusbaum_small-systems_2002 )

Read the complete text