Emergence of Functionally Differentiated Structures via Mutual Information Optimization in Recurrent Neural Networks

Yuki Tomoda, Ichiro Tsuda, Yutaka Yamaguti · 2025

Read Emergence of Functionally Differentiated Structures via Mutual Information Optimization in Recurrent Neural Networks by Yuki Tomoda, Ichiro Tsuda, Yutaka Yamaguti (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12858.txt.

Introduction

Yuki Tomoda, Ichiro Tsuda, and Yutaka Yamaguti present a study of how networks of artificial nerve cells can split into parts with distinct jobs. Their method trains a recurrent neural network while pushing down the shared information between groups of cells. The shared information is estimated with a neural tool, and the network is tested on two tasks.

One task asks the network to hold two bits in memory, and the other asks it to separate two chaotic signals. The team found that the trained networks still did the tasks well while their cells formed clear functional groups. Groups defined by how cells act together appeared before groups defined by the strength of the links between cells. The authors suggest that cells first take on special roles, and that this step then drives the network to rewire itself.

Catalog record

CREATORYuki Tomoda, Ichiro Tsuda, Yutaka Yamaguti
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE71 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-12858

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Emergence of Functionally Differentiated Structures via Mutual Information Optimization in Recurrent Neural Networks Yuki Tomoda, Ichiro Tsuda, Yutaka Yamaguti Abstract Functional differentiation in the brain emerges as distinct regions specialize and is key to understanding brain function as a complex system. Previous research has modeled this process using artificial neural networks with specific constraints. Here, we propose a novel approach that induces functional differentiation in recurrent neural networks by minimizing mutual information between neural subgroups via mutual information neural estimation. We apply our method to a 2-bit working memory task and a chaotic signal separation task involving Lorenz and Rössler time series. Analysis of network performance, correlation patterns, and weight matrices reveals that mutual information minimization yields high task performance alongside clear functional modularity and moderate structural modularity. Importantly, our results show that functional differentiation, which is measured through correlation structures, emerges earlier than structural modularity defined by synaptic weights. This suggests that functional specialization precedes and probably drives structural reorganization within developing neural networks. Our findings provide new insights into how information-theoretic principles may govern the emergence of specialized functions and modular structures during artificial and biological brain development. \unnumbered keywords functional differentiation, self-organization with constraints, mutual information, recurrent neural networks, modularity, information theory 1 Introduction During brain development, populations of neurons with specific functions organize into different regions, forming a functional map, in a process called functional differentiation ( Brodmann 1909 ) . Studies analyzing finer structural division called parcellation have revealed that these functional areas are dynamically reorganized according to specific tasks ( Glasser et al. 2016 ) . Throughout these developmental processes, modular structures consistently emerge in the brain’s organization ( Felleman and Van Essen 1991 ; Glasser et al. 2016 ; Sporns and Betzel 2016 ; Sporns 2016 ) . However, the underlying principles driving the formation of these differentiated structures remain largely unknown ( Clune et al. 2013 ; Sporns and Betzel 2016 ) . Thus, it is particularly valuable to study these problems to understand the mechanisms of emerging brain functions. Mathematical modeling offers a valuable approach to formulate these problems quantitatively and test hypotheses systematically ( Kaneko and Tsuda 2001 ) . Functional differentiation in the brain is also believed to be fundamental to the emergence of intelligence ( Kashtan and Alon 2005 ; Sporns and Betzel 2016 ; Yang et al. 2019 ) . To investigate the underlying computational principles of this process, we need models that capture both the temporal dynamics of neural activity and the capacity for adaptive specialization. Therefore, insights from recent advances in machine learning can be particularly relevant ( Sussillo 2014 ; Yang et al. 2019 ) . Recurrent neural networks (RNNs) provide an ideal framework for investigating functional differentiation because they exhibit temporal dynamics similar to biological neural circuits but can be used in controlled experiments. RNNs process sequential information, maintain internal states, and develop functional organization through learning, which are key characteristics for studying neural differentiation. Previous research has proposed that functional differentiation can be understood within the framework of self-organization with constraints, where system components emerge under constraints that act on the entire system ( Tsuda et al. 2016 ; Tsuda et al. 2022 ) . This framework suggests that the overall development may follow certain variational principles, and thus the differentiation process is not genetically predetermined. Instead, the components emerge dynamically as the system optimizes itself under global constraints. Considering that the brain’s primary information processing function is to acquire information from the environment to produce appropriate behaviors for survival, a compelling candidate for such a universal constraint is the optimization of information flow within neural networks. Mutual information (MI) provides a quantitative measure for analyzing this information flow because it captures the statistical dependencies among different neural populations. However, calculating MI in complex neural systems presents major challenges, particularly when assessing dependencies among multiple variables in high-dimensional spaces. To implement this theoretical framework computationally, efficient estimation of MI is essential. Recent advances in machine learning have addressed the challenge of estimating MI among high-dimensional variables. Belghazi et al. 2018 proposed a method for estimating MI using differentiable neural network models, called mutual information neural estimation (MINE). This approach enables the estimation of MI and the computation of gradients with respect to model parameters or input variables. This differentiability is crucial because it allows optimization of MI through gradient-based methods. In our study, we use these techniques to investigate the hypothesis that functional differentiation emerges as a result of information flow optimization. We use RNNs to investigate the role of information flow optimization in the emergence of functionally differentiated structures,. RNNs have a long history as models of neural dynamics ( Doya and Yoshizawa 1989 ; Elman 1990 ) . Recent developments in machine learning have greatly improved the capabilities of RNNs, enabling them to model complex neural dynamics and information processing mechanisms ( Sussillo 2014 ; Song et al. 2016 ; Yang et al. 2019 ; Yamaguti and Tsuda 2021 ) . Building on these developments, our study examines how functional differentiation occurs in RNNs under the constraint of MI minimization. We designed the following complementary experimental tasks to test our hypothesis (Fig. 1 ). Figure 1: Schematics of the two experimental tasks. (A) Working memory task. The network receives brief pulses on four input channels (ON 1 , OFF 1 , ON 2 , OFF 2 ) and must maintain two independent memory bits (output values of \pm 1.0 ) based on the most recent pulse received. (B) Chaotic signal separation task. The network receives a mixed 3-dimensional input signal from Lorenz and Rössler chaotic systems and must separate it into two distinct 3-dimensional output signals (1) Working memory task : A 2-bit working memory paradigm ( Sussillo and Abbott 2009 ; Hoerzer et al. 2014 ) requiring the network to maintain two binary states independently over time. This task is particularly suited for studying functional differentiation because it naturally requires separate mechanisms for each memory bit. Under MI minimization, we hypothesize that distinct neural subgroups will specialize in maintaining different memory components. (2) Chaotic signal separation task : A signal processing task requiring separation of mixed chaotic time series (Lorenz-63 ( Lorenz 1963 ) and Rössler systems ( Rössler 1976 ) ). This task encourages functional differentiation by requiring different neural subgroups to specialize in processing distinct dynamic patterns with different temporal characteristics. These tasks were selected to represent different cognitive domains while both requiring the development of specialized neural functions that can be analyzed by our information-theoretic framework. The remainder of this paper is organized as follows. We provide a brief review of related work on functional differentiation and MI optimization in neural networks. We present the necessary background knowledge for our formalization, including RNN models and MI estimation. We describe our proposed model and methodology, detailing the experimental 2-bit working memory task and signal separation of chaotic inputs task. We present the results of our computational experiments with these two tasks and analyses of functional and structural modularity within the RNN. Finally, we discuss the implications of our findings. 2 Related studies 2.1 Driving forces for functional differentiation Understanding the process through which cell populations differentiate into

Read the complete text