Optimal packing of attractor states in neural representations

John J. Vastola · 2025

Read Optimal packing of attractor states in neural representations by John J. Vastola (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12429.txt.

Introduction

Vastola, a solo author, studies in this 2025 paper how the brain should arrange its inner states for variables like place, heading, and choice. States that often change into each other should sit close, so shifts are fast. States too close, though, blur under neural noise and lose stability.

Vastola frames the tension as an optimal packing problem, kin to packing spheres. The general problem is extremely hard. Yet when the pattern of shifts between states carries symmetry, the best layout must share that symmetry. That fact lets him solve some cases exactly. Two toy cases get full treatment: even shift odds all around, and cyclic shifts around a ring. Code for the work is public.

Catalog record

CREATORJohn J. Vastola
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12429

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimal packing of attractor states in neural representations John J. Vastola Abstract Animals’ internal states reflect variables like their position in space, orientation, decisions, and motor actions—but how should these internal states be arranged? Internal states which frequently transition between one another should be close enough that transitions can happen quickly, but not so close that neural noise significantly impacts the stability of those states, and how reliably they can be encoded and decoded. In this paper, we study the problem of striking a balance between these two concerns, which we call an ‘optimal packing’ problem since it resembles mathematical problems like sphere packing. While this problem is generally extremely difficult, we show that symmetries in environmental transition statistics imply certain symmetries of the optimal neural representations, which allows us in some cases to exactly solve for the optimal state arrangement. We focus on two toy cases: uniform transition statistics, and cyclic transition statistics. Code is available at https://github.com/john-vastola/optimal-packing-neurreps23 . † † editors: Sophia Sanborn, Christian Shewmake, Simone Azeglio, Nina Miolane † † workshop: Symmetry and Geometry in Neural Representations keywords optimization, Markov chain, neural representation, neural dynamics, sphere packing, symmetry 1 Introduction Animals’ internal states appear to reflect environmental variables, like their position in space and orientation relative to some reference ( Barry and Burgess, 2014 ; Hulse and Jayaraman, 2020 ) , as well as their interactions with the environment, like decisions ( Gold and Shadlen, 2007 ) and motor actions ( Cisek, 2005 ) . As an animal acts in its environment, it must constantly update these internal states to reflect environmental changes and the results of internal computations; however, these updates cannot be instantaneous , since biophysical limitations force internal quantities to change in a somewhat continuous fashion. They are also not error-free due to noise in encoding, decoding, and neural dynamics ( Faisal et al., 2008 ; van Vreeswijk and Sompolinsky, 1996 ) . How should internal states be arranged? On the one hand, an animal can act more quickly if the next relevant internal state is ‘near’ the current one, since it can be reached more quickly. This suggests that the structure of neural representations should reflect the structure of environmental transitions; this is consistent with what is known about circuits like the head direction system, whose latent geometry mirrors the circular nature of the variable it tracks ( Ajabi et al., 2023 ) , and theoretical ideas about smoothness as a constraint on neural codes ( Stringer et al., 2019 ) . On the other hand, the closer all internal states are to one another, the easier it is for neural noise to cause problems, either via noise-induced transitions ( Burak and Fiete, 2012 ) or by increasing the likelihood of encoding and decoding errors. In principle, a ‘good’ arrangement of internal states strikes a balance between these two concerns: internal states must be packed closely enough that desired transitions can happen quickly, but not so closely that errors are likely. Given that noise sets an effective length scale for separating internal states, this issue in some ways mathematically resembles an optimal packing problem. While often quite difficult, problems like optimal sphere packing ( Zong, 2008 ) are made substantially easier to solve and understand by exploiting symmetry-related considerations. For example, Viazovska et al.’s solution of the sphere packing problem in dimensions 8 and 24 ( Viazovska, 2017 ; Cohn et al., 2017 ) crucially uses symmetry properties of the E_{8} and Leech lattices. In this paper, we attempt to formulate a toy version of the problem of constructing an ‘optimal packing’ of neural representations, and similarly turn to symmetry-related tools in order to say something meaningful about it. The particular symmetry-related claim we will motivate, and then use, is that an attractor-based neural representation of a Markov chain that exhibits a symmetry may also exhibit that symmetry. 2 Mathematical formulation of optimal packing problem To formalize our optimal packing problem, we need five things: a model of environment state statistics, a model of internal state transition dynamics, an encoding model, a decoding model, and a cost function. Environment dynamics. We will model the environment as a Markov chain on M states. In particular, we will assume that it can be characterized by a set of states \mathcal{X}=\{1,...,M\} , a base probability of state occupancy p_{0}(x) for all x\in\mathcal{X} , and a probability p(y|x) of transitioning from any state x\in\mathcal{X} to any state y\in\mathcal{X} on some characteristic time scale. Since we are interested in finding representations that respect environmental transition structure when a transition occurs , we assume without loss of generality that p(x|x)=0 for all x\in\mathcal{X} . Internal state transition dynamics. Let \mathcal{Z}=\mathbb{R}^{D} (for some D\geq M ) denote the set of all possible internal states, and assume that each environment state x\in\mathcal{X} is in one-to-one correspondence with an internal attractor state \vec{z}_{x}\in\mathcal{Z} . Assume also that the positive definite matrix \vec{\Sigma}^{-1} can be used to compute the distance D(\vec{z}_{1},\vec{z}_{2}):=\sqrt{(\vec{z}_{1}-\vec{z}_{2})^{T}\vec{\Sigma}^{-1}(\vec{z}_{1}-\vec{z}_{2})} (1) between any two internal states. The matrix \vec{\Sigma} is intended to model how noisy different directions in \mathcal{Z} are; different states are ‘closer’, in the sense of being easier to reach from one another, if the line connecting them corresponds to a particularly noisy direction. Although it is possible to write down an extremely explicit model of internal state transition dynamics, we will consider a somewhat coarse description in order to keep our problem mathematically tractable. We will assume three things: first, that transitions are essentially between attractor basins, so that the relevant quantity is the discrete distribution q(\vec{z}_{y}|\vec{z}_{x}) ; second, that there is a mechanism for destabilizing attractor states when a transition is desired, so that q(\vec{z}_{x}|\vec{z}_{x})=0 for all attractor states \vec{z}_{x} ; and third, that transitions to good approximation only depend on the distances between states. The last assumption makes sense within a landscape picture of internal dynamics (involving M attractors of similar width and depth), and can be formally justified via appealing to, e.g., Kramers’ theory ( Kramers, 1940 ; Hänggi et al., 1990 ) . Explicitly, we will assume that \begin{split}q(\vec{z}_{y}|\vec{z}_{x})&=(1-\delta_{xy})\frac{e^{-D(\vec{z}_{y},\vec{z}_{x})^{2}/2}}{Z(\vec{z}_{x})}\\ Z(\vec{z}_{x})&=\sum_{a\neq x}e^{-D(\vec{z}_{a},\vec{z}_{x})^{2}/2}\ .\end{split} (2) Encoding/decoding models. We will assume that the encoding of an environment state x\in\mathcal{X} is noisy, and that when a mistake is made, x is more likely to be encoded as a state \vec{z}_{a} near \vec{z}_{x} . Explicitly, we will assume p_{e}(\vec{z}_{a}|x)=\frac{\delta_{ax}e^{b}+(1-\delta_{ax})e^{-D(\vec{z}_{a},\vec{z}_{x})^{2}/2}}{e^{b}+Z(\vec{z}_{x})} (3) where increasing the bias b\geq 0 makes errors less likely. In the b\to\infty limit, encoding is perfect. Assuming a uniform prior, we obtain a decoding model through Bayes’ rule: p_{d}(x|\vec{z}_{a})=\frac{p_{e}(\vec{z}_{a}|x)}{\sum_{x}p_{e}(\vec{z}_{a}|x)}\ . (4) In the b\to\infty limit, since p(\vec{z}_{a}|x)=\delta_{ax} , we also have perfect decoding. Cost function. We want to penalize different possible arrangements of internal attractor states according to some objective function, so that optimizing that objective corresponds to identifying an optimal packing. ‘Optimality’ here means an arrangement which, as much as possible, produces internal dynamics (i.e., movement between attractors) whose transition statistics mirror the statistics of environmental transitions (fig:generala). The interpretation of this is that, in the absence of any external input , the internal state is poised to change in the same way that the environment is likely to change. Consider the way ring-attractor-like networks reckon with uncertainty

Read the complete text