Growing a Neural Network in Breadth, Depth, and Time

Eivinas Butkus 1,2 Kedar Garzón Gupta 1 Nikolaus Kriegeskorte 1,2 [email protected] [email protected] [email protected] 1 Columbia University 2 NSF AI Institute for Artificial and Natural Intelligence · 2026

Read Growing a Neural Network in Breadth, Depth, and Time by Eivinas Butkus 1,2 Kedar Garzón Gupta 1 Nikolaus Kriegeskorte 1,2 [email protected] [email protected] [email protected] 1 Columbia University 2 NSF AI Institute for Artificial and Natural Intelligence (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-25174.txt.

Introduction

Eivinas Butkus, Kedar Garzón Gupta, and Nikolaus Kriegeskorte train a recurrent convolutional network that can grow in width, depth, and processing time, with each resource carrying a differentiable cost. Pressures on the three resources trade off against one another. Networks expand in all directions as tasks get harder, taking extra recurrent steps when inputs are partly hidden. Strikingly, the model's processing time tracks human reaction times on an object recognition task, linking resource limits to brain-like design.

Catalog record

CREATOREivinas Butkus 1,2 Kedar Garzón Gupta 1 Nikolaus Kriegeskorte 1,2 [email protected] [email protected] [email protected] 1 Columbia University 2 NSF AI Institute for Artificial and Natural Intelligence
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36684
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.25174

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Growing a Neural Network in Breadth, Depth, and Time Eivinas Butkus 1,2 Kedar Garzón Gupta 1 Nikolaus Kriegeskorte 1,2 [email protected] [email protected] [email protected] 1 Columbia University 2 NSF AI Institute for Artificial and Natural Intelligence Abstract Spatial and temporal resource constraints are critical for both biological and artificial intelligent systems. Here we define differentiable cost terms for breadth, depth, and time within a recurrent convolutional neural network conceived as a finite subset of an infinite lattice. We optimize these costs jointly with task errors via backpropagation. We set different pressures on breadth, depth, and time, which leads to diverse computational graphs emerging organically through training. We find that all three resources can be traded off against each other to achieve a given level of accuracy. Networks grow in all three dimensions with task complexity and spontaneously take more recurrent steps when inputs are occluded. Surprisingly, time used by the model correlates with human reaction times in an object recognition task. Our framework provides a normative account of how resource constraints shape neural architectures, connecting to questions about brain design in neuroscience, and may help illuminate the diversity of neural solutions found in nature. 1 Introduction Intelligence—both biological and artificial—can be broadly characterized as the capacity to achieve goals under resource constraints [ 34 , 13 , 27 ] . Two important resources for brains and AI systems are space and time . In brains, each additional neuron adds metabolic costs, maintenance, and wiring—making a smaller brain preferable [ 8 , 21 , 38 ] . A faster brain confers advantages too [ 39 ] : failing to quickly detect a predator may lead to death. Thus, to understand a brain fully, we should consider not only the particular problems it solves (such as visual recognition), but also the particular set of spatial and temporal resource constraints it has evolved under. Engineers face analogous pressures, motivating work on model compression [ 16 , 15 ] , knowledge distillation [ 17 ] , architecture search [ 29 ] , and adaptive computation [ 14 , 35 ] . In this work, we consider three resources: breadth, depth, and time. Traditionally, these are treated as fixed hyperparameters, explored via grid search or manual tuning. Prior work has gone further: pruning methods reduce breadth and depth post-hoc [ 16 , 11 ] , network slimming learns channel importance via differentiable sparsity penalties [ 31 ] , and adaptive computation time optimizes time allocation through backpropagation [ 14 ] . However, no prior work has optimized all three jointly within a single framework. Here we define differentiable cost terms for breadth, depth, and time, and optimize those jointly with errors using backpropagation. This setup lets a network grow organically in all three dimensions, finding its own trade-off between resources and errors based on the pressures applied. To visualize this process, one can think of an infinite lattice that extends along breadth, depth, and time (Fig. 1 ). A single computational graph (model instance) is grown in this space, resulting in a unique profile of resource use. a b Figure 1 : a The space of possible computational graphs can be conceptualized as an infinite lattice, extending in the space of resource use. Here we consider breadth, depth, and time. b Each model instance is a finite subset of the infinite lattice with its own profile of resource use. Our framework lets the network select its own position in this space by optimizing differentiable resource costs. We implement the lattice using a recurrent convolutional neural network architecture with bottom-up, lateral, and top-down connections, where breadth corresponds to channels, depth to layers, and time to recurrent processing steps (Fig. 2 ). Under a given set of resource pressures, the optimization process organically prunes channels, layers, and time steps, carving out a compact subgraph from the full architecture. We train over a thousand models spanning the space of resource pressures. We find that breadth, depth, and time are fungible: shallow-wide networks can match the accuracy of deep-narrow ones, and all three resources can compensate for one another. Adaptive time allocation emerges organically—models spontaneously take more recurrent steps when inputs are occluded—and the time the model spends on individual images correlates with human reaction times, despite never being trained on human data. Networks grow in all three dimensions with task complexity, using more resources for more complex datasets. Our main contributions are: 1. A differentiable, modular, and extensible multi-resource cost (MRC) framework that jointly optimizes breadth, depth, and time costs alongside task errors via backpropagation. 2. The first joint exploration of trade-offs between breadth, depth, and time, showing that all three resources are fungible. 3. The finding that adaptive time allocation emerges organically and correlates with human reaction times, linking computational resource optimization to human perception. Our framework enables efficient exploration of the space of possible architectures, and may help illuminate the diversity of neural solutions found in nature. 2 Related work The idea that intelligent systems operate under resource constraints has a long history. Simon’s bounded rationality [ 34 ] proposed that decision-makers optimize within cognitive limits, an idea formalized more recently as computational rationality [ 13 ] and resource rationality [ 27 ] . Symbolic and probabilistic computational models have explored how resource-rational agents perceive and make decisions [ 40 , 18 , 3 ] . These works study resource constraints at the cognitive level. We impose them at the level of neural architecture (on the wiring and dynamics of the network itself). In computational neuroscience, spatial resource constraints have been studied through the lens of wiring economy—the principle that neural circuits minimize wiring costs [ 8 , 6 , 1 ] . Recent work has imposed spatial constraints on neural network models to better account for the topographic organization of visual cortex [ 28 , 4 , 32 ] . On the temporal side, recurrent neural networks have been used to study the role of recurrent processing in biological vision, showing that recurrence helps explain the dynamics of human object recognition [ 37 , 19 , 36 ] . Our work extends this line of research by jointly considering spatial and temporal resource costs within a single framework. In deep learning, prior work has trained models with adaptive depth [ 7 ] , breadth [ 31 ] , and time [ 14 ] through backpropagation. Pruning methods reduce network size post-hoc by removing units that contribute least to performance [ 25 , 16 , 26 , 11 ] , though effective pruning typically requires iterative cycles of removal and fine-tuning. Neural architecture search methods explore the space of possible architectures [ 29 ] , including differentiable approaches that optimize architecture parameters via gradient descent [ 30 ] . However, these methods search over discrete architectural choices (e.g., which operations to use), rather than imposing continuous resource costs on a fixed computational graph. Our work is the first to jointly optimize differentiable costs for breadth, depth, and time within a single framework. 3 Methods 3.1 Multi-resource cost (MRC) optimization A typical loss for a neural network includes only task performance (e.g., cross-entropy) and regularization (e.g., weight decay). Our approach adds differentiable cost terms for multiple resources and studies

Read the complete text