Identifying structural design principles shaping the computational abilities of recurrent neural networks

Tom Talpir, Elad Schneidman · 2026

Read Identifying structural design principles shaping the computational abilities of recurrent neural networks by Tom Talpir, Elad Schneidman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-23874.txt.

Introduction

Talpir and Schneidman hunt for structural features that make a recurrent neural network computationally powerful, by training thousands of networks on thousands of logic problems, in a 2026 paper on neural computation. The authors train a large collection of recurrent networks, each wired differently, able as an ensemble to compute Boolean functions from a large set. For small networks they build complete catalogs of which architecture computes which function.

The catalogs show capacity varies enormously: most architectures perform poorly, and most functions are hard. Networks containing short local cycles, loops of two or three neurons feeding back, are far more capable, and often the smallest able to solve a given function. In large networks, adding a few biologically inspired interneurons dramatically boosts capacity.

Catalog record

CREATORTom Talpir, Elad Schneidman
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.23874

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Identifying structural design principles shaping the computational abilities of recurrent neural networks Tom Talpir, Elad Schneidman Abstract Understanding how the architecture of neural networks shapes the computations they carry is a central challenge in neuroscience and machine learning. While specific circuit architectures have been linked to particular network computations and theoretical bounds on expressivity of broad classes of networks have been found, we are still missing general principles connecting the structure of finite networks to their computational capabilities. Here, we characterize the computational abilities of recurrent neural networks as a function of their connectivity by training a large collection of different networks to compute a large set of Boolean functions. For small networks, we constructed the complete “catalogs” of network-function performance, which revealed that computational capacity varies widely across architectures and that most networks show poor performance, and most functions are hard to compute. However, we show that having local 2- and 3-cycles in a network strongly enhances its computational ability, and networks with such cycles are often the minimal architectures that can solve particular functions. We further show that a small set of structural statistics accurately predict networks’ performance. Extending our analysis to large networks showed that typical networks fail even to approximate a randomly selected function. Surprisingly, adding a small number of sparsely connected biologically-inspired interneurons to the network dramatically increases computational capacity. As in small networks, adding short cycles improved networks’ capacity, outperforming acyclic or reachability-matched controls. Thus, our results identify local cycles as design principles linking neural connectivity to computational power, and offer a general framework to explore structure-function relations in computing networks. \DeclareSourcemap \maps [datatype=bibtex] \map \step [fieldset=abstract, null] \step [fieldset=language, null] \step [fieldset=urldate, null] \step [fieldset=issn, null] \step [fieldset=copyright, null] \step [fieldset=shorttitle, null] \step [fieldset=keywords, null] \step [fieldset=note, null] \step [fieldset=month, null] \step [fieldset=url, null] 1 Introduction The architectures of real-world networks often exhibit structures that reflect their functional design or their development. For example, power-law or log-normal degree distributions are commonly found in biological, social, or linguistic networks, conferring robustness to errors and failures [ 4 , 2 ] . These structures can be explained by different, relatively simple, developmental processes, such as preferential attachment, spatial branching and competition [ 27 , 26 ] . Similarly, small sub-network “motifs” are much more frequent than expected by chance, in biological and engineered networks [ 41 ] , and have been associated with specific computational roles [ 39 ] . The remarkable computations carried by biological neural networks [ 20 , 40 ] and artificial ones [ 34 , 25 ] , and their ability to learn in supervised [ 31 ] , unsupervised [ 17 ] , or reinforcement learning [ 52 , 42 ] makes them particularly appealing systems to study the relations between the structure of networks and their computational ability. The reconstructions of detailed connectivity maps of neural circuits, or “connectomes” [ 13 , 8 , 60 ] now enables the exploration of the relationship between the fine structure of neural circuits and the function that they may perform [ 5 , 35 ] . While in a few cases the relations between architecture and function have been mapped and understood [ 29 , 37 , 7 ] , most circuits do not have clear structural design or symmetries that reflect function, and while simulating particular connectomes may reflect on circuits’ function in specialized cases [ 58 , 32 ] , these do not offer a general understanding of these mappings. Surprisingly, generative models of neural circuit architectures suggest that they may be explained by a small number of biological and physical features [ 6 , 18 , 19 , 49 ] , but it is unclear how these building blocks affect which computational tasks a network can learn to perform, or how certain structural properties enhance computational performance. Compared to the typically recurrent and sparse architectures of biological neural networks, and their energetic efficiency, artificial neural networks, that were inspired by real neural networks, are typically very different. In particular, many of these networks rely on highly feed-forward architectures, and their key structural properties are the depth of the network and width of layers, the activation function of the neurons, and specialized layer structures, such as fully connected layers, convolution ones, or attention layers. Foundational theoretical work has shown that Feed-forward neural networks with exponential width can approximate any continuous function [ 22 ] , as well as lower and upper bounds on what a circuit can compute [ 21 ] , which have even been extended to some models of neural networks [ 45 ] . Moreover, depth has been shown both empirically and theoretically to play a role in determining the power of neural networks [ 15 , 12 , 31 ] . Theoretical frameworks such as PAC learning [ 56 ] attempt to answer what artificial neural networks can learn to approximate. However, these results, which are mostly asymptotic in nature, do not immediately translate into how local properties of finite networks determine their power and expressivity. This is particularly true for recurrent networks, which are prevalent in many neural circuits [ 36 , 14 ] , and while their computational power has been explored both theoretically and empirically [ 38 , 54 , 55 ] , these approaches characterize computation in terms of dynamics, not connectivity structure. For a particular class of neurons, namely combinatorial inhibitory threshold-linear networks, the nature of network dynamics can be proven to be predicted from the network’s structure [ 10 , 11 ] . However, these results only focus on the nature of network dynamics, without addressing the challenge of learning detailed computational tasks, and are yet to be generalized to other classes of neurons and networks. Analysis of spiking neural network models have identified simple architectural features that predict the functional similarity of network architectures [ 18 ] . Ideally, we would like to combine such approaches to form a general prediction of function from structure. While extensive work in graph theory and network science has characterized how the structure of a network shapes information flow, robustness, and the dynamics that unfold on networks [ 1 , 43 ] , the question of which computations a specific graph topology can actually carry has received less attention. We take a directed approach and investigate how the connectivity of recurrent neural networks shapes the computational tasks they can perform — by training a large ensemble of network architectures on a large and rich set of computational tasks and evaluating their performance (Fig. 1 ). The space of possible tasks is both vast and hard to define precisely [ 53 ] , and so we focus on Boolean functions as a comprehensive task space: they are diverse enough to capture a wide range of computational demands, and the ability to learn arbitrary Boolean functions provides a stringent test of network capacity, including the ability to fit effectively random structure. Because the number of different connectivity graphs for N neurons and the number of Boolean functions on N bits grows exponentially with N , exhaustive analysis is feasible only for small networks, where we enumerate all architectures and tasks. For larger networks, we resort

Read the complete text