Optimal Depth of Neural Networks
Qian Qi ?? · 2025
Read Optimal Depth of Neural Networks by Qian Qi ?? (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-16862.txt.
Introduction
Qian Qi asks how deep a neural network should be in a 2025 paper. Depth is usually settled by costly trial and error. Qi recasts the forward pass of a residual network as an optimal stopping problem: at each layer, the system may halt and predict, or pay for one more layer of refinement. That lens captures the trade between accuracy and compute directly.
Under a plausible rule that extra layers yield shrinking gains, Qi proves the best stopping depth is finite, even with no cap imposed. A new penalty term then nudges networks to form early-exit-friendly features. ImageNet tests confirmed the penalty works as the theory predicts. The frame also stretches to transformer models and to continuous-depth versions through free-boundary math.
Catalog record
| CREATOR | Qian Qi ?? |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-16862
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimal Depth of Neural Networks Qian Qi ?? Abstract Determining the optimal depth of a neural network is a fundamental yet challenging problem, typically resolved through resource-intensive experimentation. This paper introduces a formal theoretical framework to address this question by recasting the forward pass of a deep network, specifically a Residual Network (ResNet), as an optimal stopping problem. We model the layer-by-layer evolution of hidden representations as a sequential decision process where, at each layer, a choice is made between halting computation to make a prediction or continuing to a deeper layer for a potentially more refined representation. This formulation captures the intrinsic trade-off between accuracy and computational cost. Our primary theoretical contribution is a proof that, under a plausible condition of diminishing returns on the residual functions, the expected optimal stopping depth is provably finite, even in an infinite-horizon setting. We leverage this insight to propose a novel and practical regularization term, ℒ depth \mathcal{L}_{\text{depth}} , that encourages the network to learn representations amenable to efficient, early exiting. We demonstrate the generality of our framework by extending it to the Transformer architecture and exploring its connection to continuous-depth models via free-boundary problems. Empirical validation on ImageNet confirms that our regularizer successfully induces the theoretically predicted behavior, leading to significant gains in computational efficiency without compromising, and in some cases improving, final model accuracy. keywords [class=AMS] Primary 62L15 Secondary 68T07 , 60G40 keywords Optimal Stopping , Deep Learning Theory , Residual Networks , Sequential Decision Processes , Regularization , Dynamic Programming , Hamilton-Jacobi-Bellman Equation Peking University Beijing, 100871, P.R. China ?? 1 Introduction Deep Residual Networks (ResNets, see He et al. 2016 ) marked a pivotal moment in the history of deep learning. By introducing identity shortcut connections, ResNets overcame the notorious vanishing gradient problem, allowing for the stable training of networks with hundreds or even thousands of layers. This architectural innovation led to state-of-the-art performance on a wide array of computer vision tasks and has since been adapted for numerous other domains. Despite their empirical success, a deep theoretical understanding of ResNets remains elusive (e.g., Weinan, Han and Li 2019 ; Sirignano and Spiliopoulos 2022 ; Li, Lin and Shen 2022 ; Qi 2025 ). Key architectural decisions, most notably the network’s depth, are typically determined through extensive experimentation and resource-intensive architecture search. This process is more art than science, guided by the general principle that "deeper is better." However, this principle eventually yields diminishing returns and incurs substantial, often prohibitive, computational overhead in both training and inference. This raises a fundamental theoretical question: Is there a principled way to determine the optimal depth of a ResNet for a given problem? In this paper, we propose to answer this question by reframing the forward pass of a ResNet through the lens of classical optimal stopping theory. We model the sequence of hidden representations, from one layer to the next, as the evolution of a state in a sequential decision process. At each layer l , an agent must make a decision: 1. STOP: Halt the forward pass, using the current representation \mathbf{h}_{l} to make a prediction. This action yields a reward based on the quality of \mathbf{h}_{l} but incurs a cumulative computational cost proportional to l . 2. CONTINUE: Pass the representation to the next layer, \mathbf{h}_{l+1} , in the hope of achieving a higher-quality representation and thus a greater final reward, at the expense of an additional computational cost. This formulation naturally captures the trade-off between accuracy and computational complexity inherent in deep networks. The goal is to find a stopping rule , a policy that specifies at which layer to stop for any given input, that maximizes the expected net reward. The reward is measured by a conceptual utility function, g , which defines the semantic value of a representation at any given layer. Our framework shows that if we define the network’s goal in this way, its architecture can be analyzed as a normative model of efficient computation. 1.1 Our Contributions We formalize this intuition by defining a finite-horizon optimal stopping problem for a ResNet of maximal depth L . Our main contributions are: • We construct a formal mapping from the ResNet forward pass to a Markovian sequential decision process, defining the state space, reward function, and filtration. • We prove that under a mild and formally stated condition on the residual functions, specifically, that their expected impact diminishes with depth, the expected optimal stopping depth is provably finite, even in an infinite-horizon setting. This provides a formal basis for the phenomenon of diminishing returns observed empirically. • We leverage our theoretical framework to propose a novel regularization term, \mathcal{L}_{\text{depth}} , that can be incorporated into the standard training objective. This regularizer encourages the network to emulate our normative model by learning residual blocks that progressively "do less work," thereby yielding representations amenable to efficient, early exiting. • We demonstrate the generality of our framework by extending it to the Transformer architecture and exploring its connection to continuous-depth models and free-boundary problems. • We provide comprehensive empirical validation on ImageNet, showing that our proposed regularizer induces the theoretically predicted behavior and leads to measurable gains in computational efficiency. Our work provides a new theoretical foundation for analyzing deep architectures and offers a principled explanation for the phenomenon of diminishing returns with depth. It bridges the gap between the empirical art of network design and the rigorous mathematics of sequential decision-making. 1.2 Related Work The idea of dynamic, input-dependent computation is not new. Early exiting methods, Teerapittayanon, McDanel and Kung 2016 augment networks with intermediate classifiers. Stochastic Depth Huang et al. 2016 randomly drops layers during training, which can be seen as a randomized policy over network depth. Our work differs fundamentally from these. While they modify the architecture or training process to enforce efficiency, we provide a theoretical lens to analyze the standard, unmodified ResNet architecture itself. Our framework is also complementary to other major theoretical approaches for understanding ResNets. For instance, extensive research has analyzed the loss landscape, demonstrating that identity connections smooth the optimization surface and remove bad local minima, thus explaining their remarkable trainability Li et al. 2018 . Another prominent line of inquiry, Neural Tangent Kernel (NTK) theory Jacot, Gabriel and Hongler 2018 , explains the optimization and generalization of infinitely wide networks. While these theories provide profound insights into why deep networks can be effectively trained, they do not directly address the efficiency of computation or the optimal allocation of resources (i.e., depth) at inference time. Our approach is orthogonal: we focus not on trainability but on defining a normative principle for efficient inference. The continuous-depth limit of ResNets has been elegantly modeled as an Ordinary Differential Equation (ODE) Chen et al. 2018 , offering a different but related mathematical perspective. The Neural ODE framework views the forward pass as a continuous evolution, whereas our work models it as a discrete-time sequential decision problem, focusing on the explicit trade-off between cost and accuracy at each layer. This focus on the discrete decision process is the key distinction from the continuous evolution perspective. In Section 2 , we review the necessary preliminaries on Residual Networks and the theory of optimal stopping.