Incorporating Visual Cortical Lateral Connection Properties into CNN: Recurrent Activation and Excitatory-Inhibitory Separation

Jin Hyun Park, Cheng Zhang, Yoonsuck Choe · 2025

Read Incorporating Visual Cortical Lateral Connection Properties into CNN: Recurrent Activation and Excitatory-Inhibitory Separation by Jin Hyun Park, Cheng Zhang, Yoonsuck Choe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15460.txt.

Introduction

Jin Hyun Park, Cheng Zhang, and Yoonsuck Choe add a missing piece of the brain to vision models. Standard vision networks copy the paths that carry signals into the visual cortex. They skip the side links that join points within each visual area. Those side links are a real part of how mammals see. The team ask what changes if a network gains them.

They show that a network that feeds its own output back, with shared weights, acts like side links. A custom training rule then splits links into go and stop types, like the brain's two kinds. The changes raise accuracy on image sorting tasks. The trained model's traits also match traits seen in the real visual system.

Catalog record

CREATORJin Hyun Park, Cheng Zhang, Yoonsuck Choe
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-15460

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Incorporating Visual Cortical Lateral Connection Properties into CNN: Recurrent Activation and Excitatory-Inhibitory Separation Jin Hyun Park, Cheng Zhang, Yoonsuck Choe Abstract The original Convolutional Neural Networks (CNNs) and their modern updates such as the ResNet are heavily inspired by the mammalian visual system. These models include afferent connections (retina and LGN to the visual cortex) and long-range projections (connections across different visual cortical areas). However, in the mammalian visual system, there are connections within each visual cortical area, known as lateral (or horizontal) connections. These would roughly correspond to connections within CNN feature maps, and this important architectural feature is missing in current CNN models. In this paper, we present how such lateral connections can be modeled within the standard CNN framework, and test its benefits and analyze its emergent properties in relation to the biological visual system. We will focus on two main architectural features of lateral connections: (1) recurrent activation and (2) separation of excitatory and inhibitory connections. We show that recurrent CNN using weight sharing is equivalent to lateral connections, and propose a custom loss function to separate excitatory and inhibitory weights. The addition of these two leads to increased classification accuracy, and importantly, the activation properties and connection properties of the resulting model show properties similar to those observed in the biological visual system. We expect our approach to help align CNN closer to its biological counterpart and better understand the principles of visual cortical computation. 1 Introduction Biologically motivated neural networks for visual processing such as Neocognitron [ Fukushima, 1980 ] , Convolutional Neural Networks (CNNs) [ LeCun et al., 1989 ] , and HMAX [ Riesenhuber and Poggio, 1999 ] , drew inspiration from Hubel and Wiesel’s works on the primary visual cortical neurons [ Hubel and Wiesel, 1959 ] and subsequent developments in the field. A common feature in these models is that the alternating layers of simple cells and complex cells form a feed-forward hierarchy, starting with the afferent connections from the input (for CNN, the convolutional layers and pooling layers may serve the same purpose [ Lindsay, 2021 ] ). The hierarchy in these models loosely mimic the projections among different cortical areas in the visual pathway [ Felleman and Van Essen, 1991 ] , where each convolutional layer correspond to a distinct visual cortical area, and the connections serving as the long-range projections. Functionally, feature representations in CNN also seem to show close similarity to those in the ventral visual pathway [ Zeiler, 2014 ] . There is a major shortcoming in this, since the various visual cortical areas do not form a strict hierarchy, as there are feedback connections between the visual areas forming a recurrent loop [ Briggs, 2020 ] . Some architectural features in modern CNN variants may serve this purpose. For instance, Liao and Poggio [ Liao and Poggio, 2016 ] proposed that skipped connections in the ResNet [ He et al., 2016 ] can be seen as implementing such recurrent projections (also see Recurrent CNN: [ Liang and Hu, 2015 ] ). Figure 1: Lateral connections in the primary visual cortex (V1) of the tree shrew. The color indicates the orientation preference of the neurons, measured through optical imaging. In A and B, the anterograde tracer biocytin was injected in the neurons marked white, and their projections are shown as black. In both, we can see that the source and the target region have the same orientation preference (in A, cyan-green, and in B, red). Adapted from [ Bosking et al., 1997 ] . So far, we saw that the original and modern CNN variants faithfully incorporate the afferent connections (input to the first conv layer) and long-range projections (one conv layer to the next conv layer). However, there is yet another kind of connection that is not included in current CNN architectures: the lateral (or horizontal) connections [ Gilbert et al., 1990 ] , connections within a specific cortical area (Fig. 1 ). In a sense, these lateral connection are like connections within and across featuremaps in the same convolutional layer in CNN. If such connections are implemented, what computational role could they play? In this paper, we propose to answer the above question, by incorporating lateral connections into the CNN architecture, and test the performance and analyze the response and connection properties. This is the main novelty of our paper. Specifically, we will focus on two properties of lateral connections: (1) recurrent activation, and (2) separation of excitatory and inhibitory connections. For (1), we will model the lateral connections among the featuremaps within the same convolutional layer using shared afferent weights and unfolding through shared lateral weights, and for (2), we will design custom loss functions to force excitatory or inhibitory weights over different lateral connection bundles. Our experiments with four benchmark data sets show improved performance, and activation and connection properties similar to those found in the biological counterpart. 2 Background and Related Works The first lateral connection property we will consider is recurrent activation through these connections. There is an extensive body of research on the role of recurrent connections in visual processing [ Kar et al., 2019 , Linsley et al., 2020 , Kubilius et al., 2019 ] . Furthermore, studies have shown that incorporating such biologically motivated recurrence into CNNs often leads to performance improvements over feedforward models [ Liang and Hu, 2015 , Spoerer et al., 2017 ] . However, these models did not treat the recurrent connections in the context of lateral connections within each visual cortical area, thus they missed the opportunity to draw parallels with the rich response properties and connection properties found in laterally connected, biologically motivated visual cortical models (e.g. [ Miikkulainen et al., 2006 ] ). For example, these properties include sparsification of neural response through successive recurrent activation, and the specificity of lateral connections preferring neurons with similar orientation preference (Fig. 1 ). The second lateral connection property to be investigated is inspired by the separate excitatory and inhibitory connections found in the lateral connections of V1. The study of excitatory and inhibitory connections began in the early days of neuroscience, starting with Dale’s law. Dale’s law states that each neuron can only secrete one type of neurotransmitter, thus it can only be excitatory (glutamate) or inhibitory (GABA) but not both [ Dale, 1935 ] . Recently, artificial neural networks complying with Dale’s law have been proposed with various architectures (feedforward, recurrent, and convolutional) [ Li et al., 2024 , Cornford et al., 2020 , Blauch et al., 2022 , Xiao et al., 2018 , Liao et al., 2016 ] . However, these models employ hard constraints to enforce Dale’s law: strictly positive and negative weight matrices [ Li et al., 2024 ] , strictly non-negative synaptic weights [ Cornford et al., 2020 ] , dedicated excitatory/inhibitory outgoing sheets combined with layer normalization for stability [ Blauch et al., 2022 ] , and sign-constraints on weight matrices [ Xiao et al., 2018 , Liao et al., 2016 ] . On the contrary, our model differs in several key aspects: (1) it is loosely inspired by Dale’s law: it does not impose hard constraints on weight matrices, (2) it requires no normalization

Read the complete text