Nonlinear dynamics of localization in neural receptive fields

Leon Lufkin, Andrew Saxe, Erin Grant · 2025

Read Nonlinear dynamics of localization in neural receptive fields by Leon Lufkin, Andrew Saxe, Erin Grant (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17284.txt.

Introduction

Leon Lufkin and co-authors ask how neurons in early sensory brain areas come to respond to small, local patches of the world. Such tuned cells are common in the mammalian brain, and earlier learning models reproduced them only by building in rules that reward sparse or independent signals. Real early learning may have no such top-down rule. The team instead trains a simple feedforward network on data modeled on natural images and studies a single nonlinear neuron in detail. They derive equations for how its tuning changes during learning.

The math shows that higher-order patterns in the input data, beyond simple correlations, push the neuron toward a local receptive field. The same predictions held when many neurons learned together. The result offers a different explanation for a classic brain feature: local tuning can emerge from the dynamics of learning itself, with no efficiency rule imposed from above.

Catalog record

CREATORLeon Lufkin, Andrew Saxe, Erin Grant
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE65 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-17284

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Nonlinear dynamics of localization in neural receptive fields Leon Lufkin, Andrew Saxe, Erin Grant Abstract Localized receptive fields---neurons that are selective for certain contiguous spatiotemporal features of their input---populate early sensory regions of the mammalian brain. Unsupervised learning algorithms that optimize explicit sparsity or independence criteria replicate features of these localized receptive fields, but fail to explain directly how localization arises through learning without efficient coding, as occurs in early layers of deep neural networks and might occur in early sensory regions of biological systems. We consider an alternative model in which localized receptive fields emerge without explicit top-down efficiency constraints---a feedforward neural network trained on a data model inspired by the structure of natural images. Previous work identified the importance of non-Gaussian statistics to localization in this setting but left open questions about the mechanisms driving dynamical emergence. We address these questions by deriving the effective learning dynamics for a single nonlinear neuron, making precise how higher-order statistical properties of the input data drive emergent localization, and we demonstrate that the predictions of these effective dynamics extend to the many-neuron setting. Our analysis provides an alternative explanation for the ubiquity of localization as resulting from the nonlinear dynamics of learning in neural circuits. 1 1 1 Code to replicate experiments and figures at https://github.com/leonlufkin/localization . 1 Introduction A striking feature of peripheral responses in the animal nervous system is localization —that is, the linear receptive fields of simple-cell neurons often respond to contiguous regions much smaller than their full input domain. In vision, retinal ganglion cells approximate localized center-surround filters that tile the input ( Dac+, 00 ; Doi+, 12 ; KK, 78 ) , and simple cells downstream in primary visual cortex have localized filters that are selective for spatial frequency and orientation ( HW, 59 ; HW, 68 ; RT, 95 ; NS, 08 ; WMG, 11 ; RSH, 02 ; Rin, 02 ) . In primary somatosensory cortex, neurons respond to stimulation of restricted regions of skin ( Cro+, 11 ) and in primary auditory cortex, spatiotemporal receptive fields are typically localized in both time and frequency domains ( DWZ, 03 ; HDZ, 08 ) ; see Fig. 1 (left). By contrast, artificial learning systems do not always learn localized filters. Principal component analysis tends to fit weights that span the entire input signal, as do unregularized autoencoder neural network architectures and restricted Boltzmann machines ( Sax+, 11 ) . This difference has prompted the search for artificial learning models that can learn localized receptive fields from naturalistic stimuli, the most notable of which are sparse coding ( OF, 96 ; OF, 97 ) and independent component analysis (ICA; BS, 97 ; vHvdS, 98 ) . Sparse coding, ICA, and related compression methods that produce localized receptive fields from naturalistic data share a top-down approach—they find an efficient representation of the input signal by optimizing an explicit sparsity criterion, or an independence criterion that necessitates sparsity in a critically parameterized regime Fie, 99 ; Sax+, 11 . Though sparsity is appealing as a potentially unifying explanation for localization, localization also emerges naturally in networks trained to perform classification tasks without any explicit sparsity regularization ( KSH, 12 ; ZF, 13 ; Yos+, 15 ; Sen+, 18 ) ; see Fig. 1 (center) for an example. IG (22) distilled such examples of emergent localization by demonstrating that localized receptive fields emerge in simple feedforward neural networks trained on a data model with properties meant to approximate natural visual input, in particular, locality structure (statistical independence of non-collocated dimensions) and non-Gaussianity (higher-order cumulants are non-null). In simulations, IG (22) tie the dynamical emergence of localization to increased tuning to higher-order statistics of the input, and demonstrate that even a single neuron is sufficient to learn a localized receptive field in this setting. In this work, we build on the demonstration of IG (22) with the aim of describing the mechanisms behind the emergence of a localized receptive field in this minimal setting. The higher-order input statistics that drive localization are challenging to analyze with existing tools that exploit implied Gaussianity ( Gol+, 20 ) . By separating two stages of learning, we are able to derive equations for the effective early-time learning dynamics of the single neuron model that learns a localized receptive field from idealized naturalistic data. Our analytical model identifies a concise description of the higher-order statistics that drive emergence, and we validate both positive and negative predictions of this analytical model via simulations with many neurons; see Fig. 1 (right). These findings suggest an alternative path to account for the ubiquity of localization in early neural responses as resulting from the interaction of the nonlinear dynamics of learning in neural circuits and naturalistic data with higher-order statitistical structure, rather than an explicit efficiency criterion. 1-real.pd 5-real.pd 2-real.pd 6-real.pd 3-real.pd 7-real.pd 4-real.pd 8-real.pd ( Rin, 02 ) fig2-d.pd fig2-e.pd fig2-f.pd fig2-h.pd ( dBM, 98 ) fig2-a1.pd fig2-a2.pd fig2-a4.pd fig2-a5.pd ( Sin+, 18 ) ( KSH, 12 ) 1-2d-blob.pd 2-2d-blob.pd 3-2d-blob.pd 4-2d-blob.pd 5-2d-blob.pd 6-2d-blob.pd 7-2d-blob.pd 8-2d-blob.pd ( HO, 00 ) 1-2d-blob.pd 2-2d-blob.pd 3-2d-blob.pd 4-2d-blob.pd 5-2d-blob.pd 6-2d-blob.pd 7-2d-blob.pd 8-2d-blob.pd M1 SCM Figure 1 : (Left) Localization in spatial receptive fields (RFs) measured from non-human primate (NHP) primary visual cortex ( Rin, 02 , Fig. 2) and in spatiotemporal RFs measured from NHP ( dBM, 98 , Fig. 2) and ferret ( Sin+, 18 , Fig. 2) primary auditory cortex. (Center) Half-slice of the localized first-layer kernels of AlexNet trained for ImageNet classification ( KSH, 12 ) . (Right) Localized receptive fields learned from the task of Section 2.3 in 2-D using ICA ( HO, 00 ) and the soft committee machine (SCM; M1 with fixed second-layer weights) of Section 2.1 . Localization—spatial and/or temporal selectivity—appears across settings, as measured by response maximization in biological systems (left) and by inspecting linear filters in artificial systems (center, right). 2 Modeling approach We extend the setting of IG (22) , a minimal example of a neural network that learns localized receptive fields from idealized naturalistic data. We analyze the dynamics of learning in this setting in Section 3 and validate our analytical model with simulations in Section 4 . 2.1 Neural network architecture and learning algorithm We consider a two-layer feedforward neural network with nonlinear activation and scalar output. While simple, this architecture is highly expressive, capable of approximating arbitrary integrable univariate functions with appropriate scaling ( Bar, 93 ; Pin, 99 ) , and exhibits rich feature learning dynamics that underlie the performance of models at scale ( Woo+, 20 ) , making this architecture the ongoing subject of theoretical neural network analyses ( MMN, 18 ; Gol+, 19 ; Vei+, 22 ) . We denote a two-layer network with N -dimensional input, M hidden units, and one-dimensional scalar output as Model 1 ( many-neuron architecture ). (M1) \hat{y}(\mathbf{x})=b^{(2)}+\sum_{m=1}^{M}w_{m}^{(2)}\sigma\left(b_{m}^{(1)}+\langle\mathbf{w}_{m}^{(1)},\mathbf{x}\rangle\right) where \sigma:\R\to\R is a pointwise nonlinearity such as the rectified linear unit (ReLU) or sigmoid function, \mathbf{w}_{m}^{(1)}\in\R^{N} and w_{m}^{(2)}\in\R are learnable weights, b_{m}^{(1)},b^{(2)}\in\R are learnable bias terms, and \langle\cdot,\cdot\rangle denotes the standard Euclidean inner (dot) product on \R^{N} . When the second-layer parameters are fixed, this model is known as a soft-committee machine (SCM; SS, 95 ) , which ( IG,

Read the complete text