Paradoxical increase of capacity due to spurious overlaps in attractor networks
Marco Benedetti 1 , Nicolas Brunel 1 , Enzo Marinari 2 and Ulises Pereira-Obilinovic 3,∗ · 2025
Read Paradoxical increase of capacity due to spurious overlaps in attractor networks by Marco Benedetti 1 , Nicolas Brunel 1 , Enzo Marinari 2 and Ulises Pereira-Obilinovic 3,∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17593.txt.
Introduction
Associative memory networks store patterns in their wiring and recall them later through their collective dynamics, much like the classic Hopfield model. Marco Benedetti, Nicolas Brunel, Enzo Marinari, and Ulises Pereira Obilinovic examine a nuisance in such networks: overlaps with the wrong stored memories, called spurious overlaps, which act as noise during recall. In classic models those overlaps widen the spread of inputs to each neuron without shifting their average.
With a learning rule drawn from real neurobiological data, the picture changes. Benedetti and his co-authors show that spurious overlaps then pull the average input down, and that drop paradoxically raises how many patterns the network can store, by making its activity sparser. The result ties a detail of biologically realistic learning directly to memory capacity.
Catalog record
| CREATOR | Marco Benedetti 1 , Nicolas Brunel 1 , Enzo Marinari 2 and Ulises Pereira-Obilinovic 3,∗ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-17593
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Paradoxical increase of capacity due to spurious overlaps in attractor networks Preprint: APS/123-QED Marco Benedetti 1 , Nicolas Brunel 1 , Enzo Marinari 2 and Ulises Pereira-Obilinovic 3,∗ Abstract In Hopfield-type associative memory models, memories are stored in the connectivity matrix and can be retrieved subsequently thanks to the collective dynamics of the network. In these models, the retrieval of a particular memory can be hampered by overlaps between the network state and other memories, termed ‘spurious overlaps’ since these overlaps collectively introduce noise in the retrieval process. In classic models, spurious overlaps increase the variance of synaptic inputs but do not affect the mean. We show here that in models equipped with a learning rule inferred from neurobiological data, spurious overlaps collectively reduce the mean synaptic inputs to neurons, and that this mean reduction causes in turn an increase in storage capacity through a sparsening of network activity. Our paper demonstrates a link between a specific feature of experimentally inferred plasticity rules and network storage capacity. I Introduction Attractor neural networks of the Hopfield type are a popular theoretical framework for learning and memory. In these models, memories are stored as fixed-point attractors of the network thanks to synaptic plasticity. Memories can be retrieved by providing an external input that leads the network in the basin of attraction of the corresponding fixed point attractor. This framework naturally explains the experimentally observed phenomenon of persistent activity in animals performing delayed-response tasks ( Fuster and Alexander 1971 ; Miyashita 1988 ; Funahashi et al. 1989 ; Goldman-Rakic 1995 ; Inagaki et al. 2019 ; Constantinidis et al. 2018 ; Khona and Fiete 2022 ) . In classical attractor networks such as the Hopfield model Amit et al. 1985 ; Amit et al. 1985 , retrieval of a particular memory is promoted by the overlap between the network state and this specific memory, but overlaps with other memories act as noise during the retrieval process, limiting the storage capacity of the model. When analyzing networks whose learning rules and neural nonlinearities are inferred from in vivo data Lim et al. 2015 ; Pereira and Brunel 2018 , we find however a qualitatively different behavior. Although overlaps with non-retrieved memories (termed spurious overlaps ) are individually small, their collective effect contributes an order-one reduction to the mean input current. This global suppression promotes sparser retrieval states, in which only a small fraction of neurons are strongly active. The resulting sparsity, in turn, sharply enhances the storage capacity of the network. We develop a Dynamical Mean Field Theory Tirozzi and Tsodyks 1991 ; Pereira-Obilinovic et al. 2023 that captures the influence of spurious overlaps on network dynamics. Using this theory, we derive analytical expressions characterizing retrieval properties and show that spurious overlaps can potentially increase memory capacity by reducing mean input currents to neurons. We show that this reduction occurs when the function that describes the dependence of the learning on post-synaptic firing rate (thereafter referred to as ‘POST function’) has a negative average over stored patterns. In classical networks using covariance rules Sejnowski 1977 such as the Hopfield model Hopfield 1982 , this average is zero, so the effect of spurious overlaps on the mean input current vanishes. However, learning rules inferred from data have a significantly negative average of the POST function Lim et al. 2015 ; Pereira and Brunel 2018 , leading to a strong increase in memory capacity. II The model We consider a network of N analog neurons (e.g. Hopfield 1984 ; Dayan and Abbott 2005 ) whose rates evolve according to \dot{r}_{i}=-r_{i}+\Phi\Big(I_{i}+\sum_{j}c_{ij}J_{ij}r_{j}\Big). (1) In eq. 1 , c_{ij} is a random Erdos-Renyi matrix, whose statistics follows P(c_{ij}=x)=c\delta(1-x)+(1-c)\delta(x). (2) We concentrate here on the highly diluted regime c\ll 1 . A sparse, asymmetric connectivity structure implies that correlations among firing rates of different neurons can be neglected at leading order, allowing a Mean Field Theory analysis. This regime is relevant for cortical microcircuits, where each neuron receives a large number of inputs ( K\sim 10^{3} ), but connection probabilities are low, on the order of \sim 10% in cortex Mason et al. 1991 ; Markram et al. 1997 ; Holmgren et al. 2003 ; Thomson and Lamy 2007 ; Lefort et al. 2009 ; Campagnola et al. 2022 and \sim 1% in hippocampus Guzman et al. 2016 . The coupling matrix J_{ij} is shaped by presenting to the network an infinite sequence of independently and identically distributed (i.i.d.) Gaussian input patterns \xi^{\mu}_{i}\sim\mathcal{N}(0,1) . This choice is made for consistency with Lim et al. 2015 , where the learning rule was inferred from in vivo data. We assume that, during learning, inputs induce firing rates \tilde{r}_{i}^{\mu}=\Phi(\xi^{\mu}_{i}) , which in turn modify the coupling matrix according to J_{ij}^{\mu}=\Big(1-\frac{a}{Nc}\Big)J_{ij}^{\mu-1}+\frac{b}{Nc}f(\tilde{r}_{i}^{\mu})g(\tilde{r}_{j}^{\mu}), (3) where a and b are parameters of the model that describe synaptic decay, and the strength of imprinting of new memories, respectively Mézard et al. 1986 ; Pereira-Obilinovic et al. 2023 . The transfer function \Phi(\cdot) and the dependence of the learning rule on pre- ( g(\cdot) , the PRE function) and post- ( f(\cdot) , the POST function) synaptic firing rates are described by sigmoid functions \Phi(x)=r_{m}/(1+\exp[-\beta_{\Phi}(x-h_{0})] , f(x)=[2q_{f}-1+\tanh(\beta_{f}(x-x_{f}))]/2 , g(x)=[2q_{g}-1+\tanh(\beta_{g}(x-x_{g}))]/2 . Sigmoidal \Phi(x) and f(x) have been shown to provide good fits to experimental data, using r_{m}=76.2 Hz, h_{0}=2.46 , \beta_{\Phi}=0.82 , \beta_{f}=0.28 s, x_{f}=26.6 Hz, and q_{f}=0.83 . Importantly, these parameters lead to a negative average of the POST function, \int P(\xi)f\big(\Phi(\xi)\big)d\xi<0, (4) which ensures that familiar stimuli lead to lower average rates than novel ones Woloszyn and Sheinberg 2012 ; Lim et al. 2015 . For the function g(x) , consistent with Pereira and Brunel 2018 , we choose \beta_{g}=\beta_{f} s, x_{g}=x_{f} , and q_{g} is chosen so that the PRE function has a zero average over the distribution of firing rates, i.e. \int P(\xi)g\big(\Phi(\xi)\big)d\xi=0 , leading to q_{g}=0.89 . In the following, unless otherwise specified, we use the parameters that best fit the data, except x_{f}=x_{g}=22 Hz. The resulting palimpsestic coupling matrix is: J_{ij}=\frac{b}{Nc}\sum_{\mu=0}^{\infty}e^{-\frac{a\mu}{Nc}}f(\tilde{r}_{i}^{\mu})g(\tilde{r}_{j}^{\mu}), (5) with forgetting timescale \tau=1/a and learning rate b . In Eq. ( 5 ), the sum over \mu is a sum over memories with different ages, \mu=0 being the most recent one, while \mu\gg cN labels old memories that have been forgotten, due to the exponential decay term. We anticipate that the maximal age at which patterns can be retrieved scales linearly with cN Mézard et al. 1986 ; Derrida and Nadal 1987 ; Pereira-Obilinovic et al. 2023 , and define a rescaled age s as s=\mu/(cN) . III Mean Field Theory analysis Once the coupling matrix is assembled, one of the learned patterns is presented to the network in the form of an external field I_{i}=\xi^{\mu}_{i} , and the system is allowed to reach a steady state of the dynamics. Then, the external input is removed, setting I_{i}=0 , and the dynamics proceeds until the system reaches a steady state. The characteristics of this latter “retrieval state” can be understood by studying the statistics of the local fields that drive the dynamics \lambda_{i}(t):=\sum_{j}c_{ij}J_{ij}r_{j}(t). (6) As detailed in appendices A and B , in the large N limit fields \lambda_{i}(t) behave as independent random variables \lambda_{i}(t)\sim S+h_{i}(t)+e^{-\frac{a\mu}{Nc}}bf(\tilde{r}_{i}^{\mu})m^{\mu}. (7) where: • m^{\mu}:=1/N\sum_{i}g(\tilde{r}_{i}^{\mu})r_{i} is