The Depth Delusion: Why Transformers Should Be Wider, Not Deeper

Md Muhtasim Munif Fahim, Prof. Md Rezaul Karim, PhD, Md Muhtasim Munif Fahim, Prof. Md Rezaul Karim, PhD · 2026

Read The Depth Delusion: Why Transformers Should Be Wider, Not Deeper by Md Muhtasim Munif Fahim, Prof. Md Rezaul Karim, PhD, Md Muhtasim Munif Fahim, Prof. Md Rezaul Karim, PhD (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20994.txt.

Introduction

Md Muhtasim Munif Fahim and Md Rezaul Karim argue that language models should grow wide, not deep. Their 2026 paper challenges a quiet assumption in scaling laws. Scaling laws describe how loss falls as parameters and data grow. They treat architecture as interchangeable. A billion parameters could form a shallow, wide model. Or they could form a deep, narrow one. The title calls the preference for depth a delusion.

The authors compare these shapes head to head. A shallow-wide network has few layers but broad ones. A deep-narrow network stacks many thin layers. If width wins, training and serving could both get cheaper. The paper is an argument backed by this comparison. It speaks to labs deciding how to spend a fixed parameter budget.

Catalog record

CREATORMd Muhtasim Munif Fahim, Prof. Md Rezaul Karim, PhD, Md Muhtasim Munif Fahim, Prof. Md Rezaul Karim, PhD
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20994

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Depth Delusion: Why Transformers Should Be Wider, Not Deeper Md Muhtasim Munif Fahim, Prof. Md Rezaul Karim, PhD, Md Muhtasim Munif Fahim, Prof. Md Rezaul Karim, PhD Abstract Neural scaling laws describe how language model loss decreases with parameters and data, but treat architecture as interchangeable—a billion parameters could arise from a shallow-wide model (10 layers & 8,192 hidden dimension) or a deep-narrow one (80 layers & 2,048 hidden dimension). We propose architecture-conditioned scaling laws decomposing this dependence, finding that optimal depth scales as D ∗ ∝ C 0.12 D^{*}\propto C^{0.12} while optimal width scales as W ∗ ∝ C 0.34 W^{*}\propto C^{0.34} , meaning width should grow 2.8 × \times faster than depth. We discover a critical depth phenomenon: beyond D crit ∝ W 0.44 D_{\text{crit}}\propto W^{0.44} (sublinear in W W ), adding layers increases loss despite adding parameters—the Depth Delusion . Empirically, we validate these findings across 30 transformer architectures spanning 17M to 7B parameters, each trained on representative high-compute samples, achieving R 2 = 0.922 R^{2}=0.922 . Our central finding: at 7B scale, a 64-layer model (6.38B params) underperforms a 32-layer model (6.86B params) by 0.12 nats, despite being significantly deeper. This demonstrates that optimal depth-width tradeoffs persist at the production scale. (a) Capacity vs. Loss (b) Depth Delusion at W=512 Figure 1 : Primary Evidence. (a) Test loss scales predictably with parameters until depth-width limits are reached. (b) For fixed width (W=512), increasing layers beyond D_{\mathrm{crit}}=16 creates a U-shaped penalty. The partial recovery at 32L remains significantly above the 16L optimum. 1 Introduction The success of large language models rests on a remarkable empirical regularity: test loss decreases as a power law in both model size and training data ( 10 ; 8 ) . These neural scaling laws have become the foundation for billion-dollar training investments, guiding decisions from GPT-3 ( 2 ) to PaLM ( 3 ) to LLaMA ( 21 ) . Yet current scaling laws harbor a critical blind spot. They predict loss as a function of parameter count N and token count T , but remain silent on how those parameters should be arranged. Consider two models with identical parameter counts: • Model A: 10 layers \times 8,192 width \approx 8B parameters • Model B: 80 layers \times 2,900 width \approx 8B parameters Current scaling laws predict identical loss for both, yet practitioners universally choose the deeper configuration. GPT-3 uses 96 layers. PaLM uses 118. Across model generations, depth has grown faster than width, guided by the intuition that deeper networks enable more sophisticated compositional reasoning and multi-hop inference. Is this preference for depth justified? In this paper, we provide a surprising answer: beyond a critical depth, adding layers hurts performance . We call this phenomenon the Depth Delusion —the mistaken belief that more depth is always beneficial. Our work makes three main contributions: 1. Architecture-Conditioned Scaling Laws ( Section 3 ). We propose, based on gradient flow dynamics and empirical data, a theory decomposing how depth D and width W separately affect loss: • Critical Depth ( 3.3 ): A sublinear scaling limit D_{\text{crit}}\propto W^{0.44} beyond which deeper is worse. • Optimal Scaling ( 3.4 ): Our Ansatz implies W^{*} should grow 2.8 \times faster than D^{*} . 2. Large-Scale Empirical Validation ( Section 4 ). We train 30 transformer architectures spanning depths 2–80 and widths 256–6,144, with model sizes up to 7B parameters: • Our scaling law achieves R^{2}=0.922 across all architectures, including billion-scale models. • At 7B scale, a 64-layer model underperforms a 32-layer model by 0.12 nats ( Section 4.8 ). • We confirm that optimal depth scales much slower than width ( D^{*}\propto C^{0.12} ). 3. Implications for LLM Design ( Section 5 ). If our framework extrapolates beyond 7B scale , existing models like GPT-3, PaLM, and LLaMA may be 3.6–4.9 \times deeper than their theoretical critical depth, suggesting that current scaling recipes may effectively be suboptimal for compute-constrained training. 1.1 The Mechanism: Why Depth Hurts The core insight behind our results is gradient starvation . In a transformer with D layers, consider how the gradient signal propagates backward during training. At layer \ell (counting from bottom), the gradient magnitude satisfies: \|\nabla_{\ell}L\|\approx\|\nabla_{D}L\|\cdot e^{-(D-\ell)/\tau(W)} (1) where \tau(W)=c\log W is the gradient persistence length ( Figure 2 ). Figure 2 : Gradient Starvation Mechanism. Exponential decay of gradient signal through layers for different widths. Wider models support more persistence, but all exhibit a 1/e threshold defining D_{\text{crit}} . When depth D exceeds \tau(W) , the earliest layers receive exponentially weak gradients. These layers have parameters allocated to them—consuming both memory and compute—but those parameters cannot learn effectively. This creates wasted capacity : we pay the full cost of those parameters without receiving commensurate learning benefit. The surprising implication: at some point, the optimization penalty from gradient starvation exceeds the representational benefit of additional layers, causing overall performance to degrade despite increasing model capacity . This is the Depth Delusion. 2 Related Work Neural Scaling Laws. The observation that deep learning performance follows power laws dates to 7 , who showed generalization error scales as \epsilon\propto D^{-\alpha} across domains. 10 established the canonical form L\propto N^{-0.076} for language models, with loss decomposing into capacity and data terms. 8 refined this by jointly optimizing over parameters and data, deriving the “Chinchilla” recipe that N and T should scale roughly equally with compute. Crucially, all prior scaling law work treats architecture as fixed or interchangeable. 10 explicitly state “we find that the loss depends only on the total N and not on the shape.” Our work directly challenges this assumption by showing that depth and width have fundamentally different scaling exponents ( 0.12 vs. 0.34 ). Depth vs. Width Tradeoffs. The relative merits of depth versus width have been debated since the earliest days of neural networks. For fully-connected networks, 14 proved depth-separation results: certain functions require exponentially many neurons with bounded depth. For CNNs, 23 demonstrated that wide residual networks (WRN) can match or exceed very deep ResNets ( 6 ) with fewer parameters and faster training. For transformers specifically, 20 systematically compared encoder-decoder, decoder-only, and other variants, finding that depth and width matter differently for different tasks. 12 studied depth limits in residual networks, showing that effective depth is bounded by skip connection decay. However, none of this work derives quantitative scaling exponents for optimal depth and width as we do. Signal Propagation in Deep Networks. Understanding gradient flow through deep architectures is central to training stability. 15 analyzed transformers specifically, proving that stacking self-attention layers causes “rank collapse” of token representations at initialization, hindering gradient flow to early layers. 5 proved the striking result that pure self-attention (without MLP blocks) loses rank doubly exponentially with depth. Our gradient persistence length \tau(W)=c\log W connects to this literature: the \log W scaling arises from attention entropy, which 4 showed scales logarithmically with model dimension. Wider models maintain higher-entropy attention patterns, enabling gradients to persist through more layers. Efficient Transformers. Architectural innovations like ReZero ( 1 ) and depth-dependent scaling have aimed to stabilize deep training. Recent open models such as Mistral ( 9 ) demonstrate the empirical success of wider architectures (e.g., 32 layers for 7B parameters). Our

Read the complete text