CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry

Huzama Ahmad *, Cao Viet Hai Nam, Se-Young Yun · 2026

Read CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry by Huzama Ahmad *, Cao Viet Hai Nam, Se-Young Yun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26538.txt.

Introduction

Huzama Ahmad, Cao Viet Hai Nam, and Se-Young Yun ask why the deepest layers of transformer models add so little. Transformers, the standard architecture behind language models, stack identical blocks. The authors propose Gradient Fan-in Asymmetry as the explanation. In common designs, each layer's training gradient sums the direct path plus all paths through later layers. Early layers therefore receive richer gradients than late ones.

Two efficiency methods follow from that account. CascadeFormer narrows the model with depth, matching the uneven flow of information. It matched a uniform baseline's quality at lower latency and higher throughput. CascadeFlow Pruning removes layers using gradients gathered during training, with no later analysis. It beat standard pruning rules on quality and stability. Experiments on models up to just over a billion parameters supported the theory. Equalising gradient sizes did not restore deep layers, pointing to structure, not size, as the bottleneck.

Catalog record

CREATORHuzama Ahmad *, Cao Viet Hai Nam, Se-Young Yun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.26538

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CascadeFormer: Depth-Tapered Transformers Motivated by Gradient Fan-in Asymmetry Huzama Ahmad *, Cao Viet Hai Nam, Se-Young Yun Abstract Deep Transformers are composed of uniformly stacked residual blocks, yet their deepest layers often add little value. We present two efficiency methods that exploit this asymmetry. CascadeFormer tapers width with depth to match the uneven information flow across layers, achieving comparable perplexity to a uniform baseline at the same training budget while reducing latency by 8.6% and increasing throughput by 9.4%. CascadeFlow Pruning removes layers using accumulated training gradients, with no post hoc analysis. It outperforms standard heuristics on perplexity and rank-stability and stays competitive on downstream accuracy. To motivate these methods, we propose Gradient Fan-in Asymmetry (GFA) as a structural account of why deeper layers contribute less. In Pre-LayerNorm residual stacks, the gradient at a layer is the sum of an identity path and all downstream functional paths, producing a gradient fan-in that decays linearly with depth (and quadratically under deep supervision), yielding richer gradients for early layers and sparser ones for later layers. We provide correlational and interventional evidence for GFA on models trained from scratch up to 1.2B parameters. Across Transformers and ResNets, accumulated training gradients follow the theoretical fan-in and are associated with post hoc layer importance. Two interventions point to structure rather than magnitude as the bottleneck: equalizing per-layer gradient norms does not restore late-layer value, while increasing downstream path counts via parameter-shared repetition restores and elevates it. Whether gradient magnitude proxies fan-in beyond high-rank regimes, and how these dynamics behave at the 100B+ scale, remain open questions. Introduction The uniform scaling of transformer blocks Vaswani et al. 2017 , simply repeating identical layers to create deeper models, has been the driving principle behind the success of Large Language Models ( Radford et al. ; Brown et al. 2020 ; Touvron et al. 2023 ) . However, uniform stacking masks a functional asymmetry. For instance, when we evaluate a pretrained LLaMA model on WikiText, its deeper layers exhibit high representational similarity, a key indicator of redundancy ( Gromov et al. 2024 ) (Figure 1 a). This asymmetry is even more pronounced in architectures like LayerSkip ( Elhoushi et al. 2024 ) , which skips later layers by exiting the network early, revealing their sharply declining functional contribution (Figure 1 b). The conventional explanation for this phenomenon points to attenuated gradients in deeper layers Li et al. 2025 . The explanation is correct, but it may treat a symptom rather than the underlying factor. We propose that the relevant quantity is not gradient magnitude but the compositional diversity of the gradient, a structural bottleneck we term Gradient Fan-in Asymmetry (GFA) . Residual connections ( He et al. 2016a ) turn the network into an implicit ensemble of paths of varying lengths ( Veit et al. 2016 ) , so shallow layers receive gradient from all downstream blocks while the deepest layers, aggregating from few, receive a structurally simple, information-poor gradient. We support the GFA hypothesis and demonstrate its utility through a sequence of empirical arguments, all on models trained from scratch up to 1.2B parameters. First, we observe a positive correlation between per-layer gradient norms \bar{g}_{i} and eventual functional importance \Delta\mathcal{M}_{i} . We then move beyond correlation with two interventional tests: one ablative, showing that artificially amplifying late-layer gradient magnitude fails to restore late-layer importance, and one constructive, showing that structurally increasing their path counts via layer repetition does restore it. These interventions are consistent with a structural (not magnitude) bottleneck. Finally, we translate the GFA account into two practical applications: CascadeFlow Pruning (CFP) , an efficient method leveraging accumulated training gradients to outperform standard pruning heuristics, and the CascadeFormer , an architecture that tapers width with depth to align model capacity with the natural flow of compositional gradient diversity, improving inference efficiency at fixed training FLOPs. We reframe gradient magnitude as a proxy for a structural information imbalance, not as a cause to be fixed. Our contributions are: • We design the CascadeFormer , an architecture that internalizes the GFA principle. By tapering network width with depth to match the natural flow of compositional gradient diversity, it reduces latency and increases throughput over a uniform baseline with equal training FLOPs and comparable perplexity. • We introduce CascadeFlow Pruning (CFP) , an efficient method that leverages accumulated training gradients as a proxy for structural importance to prune layers, outperforming standard heuristics without requiring expensive post hoc analysis. • We propose Gradient Fan-in Asymmetry as a structural account of layer redundancy and provide correlational and interventional evidence for it on models trained from scratch up to 1.2B parameters. Two complementary interventions, an ablative test (artificially equalizing gradient norms fails to restore importance) and a constructive one (structurally increasing path counts via layer repetition succeeds), are consistent with a bottleneck rooted in the gradient’s compositional complexity rather than its raw magnitude. Whether magnitude proxies fan-in beyond high-rank regimes, and how these dynamics behave at the 100B+ scale, remain open questions. Figure 1: Deeper Transformer layers show diminishing contributions. Illustrative motivation from pretrained public checkpoints (distinct from the from-scratch models studied in the rest of the paper). (a) In a pretrained LLaMA 13B model, representational similarity across layers increases with depth, signalling growing redundancy. (b) A pretrained LayerSkip LLaMA 8B (an early-exit/deep-supervised model, where the effect is pronounced by design) makes the consequence explicit: layer importance, measured by Functional Importance ( \Delta\mathcal{M} ) upon its removal, is concentrated in the initial layers while the functional value of later layers collapses. Related Work Layer Redundancy for Model Efficiency. Deep networks exhibit layer redundancy that can be exploited for compression and faster inference ( Gromov et al. 2024 ; Sun et al. 2025 ; Chen et al. 2025 ) . Structured pruning removes entire blocks with minor loss ( Chen et al. 2023 ; Frantar and Alistarh 2023 ; Ma et al. 2023 ; Xia et al. 2024 ; Kim et al. 2024 ; He et al. 2024 ; Sun et al. 2024 ) , while training-time methods like LayerDrop ( Fan et al. 2020 ) and early-exit/skip mechanisms ( Elhoushi et al. 2024 ; Xin et al. 2020 ; Liu et al. 2020 ; Zhao et al. 2025 ; Men et al. 2025 ) allow dynamic redundancy management. These works document that redundancy exists and how to exploit it; the mechanistic accounts that do exist attribute it largely to gradient magnitude attenuation, whereas we propose a compositional (fan-in) account. The Architectural Roots of Redundancy. Residual networks ( He et al. 2016a ; He et al. 2016b ) can be viewed as implicit ensembles of shorter paths Veit et al. 2016 , a property inherited by Transformers with Pre-LayerNorm architectures ( Xiong et al. 2020 ; Touvron et al. 2023 ) . This structure often causes deeper layers to contribute minimally, leading to redundancy ( Takase et al. 2023 ; Sun et al. 2025 ; Li et al. 2025 ) . A complementary line of work shows how architectural and normalization choices regulate gradient propagation across depth ( Wang et al. 2022 ; Shleifer et al. 2021 ; Li et al. 2024 ) . Our methods target the Pre-LN setting. Building on these

Read the complete text