Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization
Haoming Meng, Anton Sugolov, Vardan Papyan · 2026
Read Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization by Haoming Meng, Anton Sugolov, Vardan Papyan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30813.txt.
Introduction
Haoming Meng, Anton Sugolov and Vardan Papyan propose smoothing the updates of a neural network across its layers during training. Networks built from repeated blocks, such as transformers, show structured links between layers as they train, yet standard optimisers update each layer on its own. Their framework, Depth-wise Gradient Augmentation, transforms the set of block updates along the depth direction before applying them. Gradient Smoothing is the concrete method, using a simple local window that averages neighbouring layers' updates.
The method wraps any base optimiser, such as SGD, Adam or Muon, at minimal extra cost. Across language model pretraining, reasoning post-training, diffusion models and image classification, it consistently improved optimisation and generalisation. It also made representations evolve more smoothly with depth.
Catalog record
| CREATOR | Haoming Meng, Anton Sugolov, Vardan Papyan |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30813
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization Haoming Meng, Anton Sugolov, Vardan Papyan Abstract Deep neural networks with repeated architectural blocks, such as transformers, often exhibit structured relationships across layers that emerge during training. Motivated by this observation, we introduce Depth-wise Gradient Augmentation , a general optimization paradigm in which the update applied to each layer is obtained by transforming the collection of block-wise optimizer updates along the depth dimension. Within this framework, we study Gradient Smoothing , a family of depth-wise smoothing methods, and instantiate it with a simple local Window Smoothing operator. The resulting method operates directly on block-wise updates produced by arbitrary base optimizers (e.g., SGD, Adam, Muon), incurs minimal computational overhead, and is compatible with existing optimization pipelines. We evaluate Gradient Smoothing across a diverse set of architectures and training regimes, including language model pretraining, RL post-training of LLMs for reasoning, diffusion modeling, and image classification with Vision Transformers. Across these settings, Gradient Smoothing consistently improves optimization and generalization performance without modifying model architectures or training objectives. We further show that it promotes more structured representation evolution across depth, consistent with its interpretation as a structured depth-wise preconditioning method. Together, these results establish Depth-wise Gradient Augmentation as a promising framework for exploiting cross-depth structure in optimization and demonstrate Gradient Smoothing as a simple and broadly applicable instantiation. † † affiliationnotice: Equal contribution 1 Introduction Many modern deep neural networks are composed of repeated architectural blocks with shared computational structure, such as Residual Networks (ResNets) ( 15 ) and transformers ( 45 ) . Recent work has shown that networks of this form often exhibit strong structural regularities across depth that emerge during training. In particular, studies on Transformer Block Coupling ( 3 ) and Residual Alignment ( 25 ) demonstrate that singular vectors of block Jacobians and residual representations become aligned across layers, suggesting a form of implicit coordination throughout depth. Related evidence appears in studies of layer similarity and pruning in large language models ( 12 ) , as well as analyses showing smooth evolution of information-theoretic quantities across depth ( 40 ) . These observations suggest that repeated blocks in deep networks do not evolve independently, but instead exhibit coordinated representational and dynamical structure across depth. However, standard optimization methods typically construct and apply block-wise optimizer updates independently (aside from their dependencies through the forward and backward passes), without explicitly leveraging relationships between layers. This naturally raises the question: Can optimizer updates themselves be augmented to exploit the structure that emerges across depth? In this work, we introduce a general paradigm of Depth-wise Gradient Augmentation . Rather than applying the optimizer update produced for each block in isolation, the update applied to each layer is obtained by applying a depth-wise augmentation operator to the collection of block-wise optimizer updates. This perspective encompasses a broad class of update transformations acting along the depth dimension. As a first instantiation of this framework, we study Gradient Smoothing , motivated by the empirical observation that neighboring layers often learn related representations and exhibit increasingly aligned optimization dynamics throughout training. We focus on a particularly simple realization, Window Smoothing , which augments each block update by averaging optimizer updates from a local window of neighboring layers. Window Smoothing is simple to implement, incurs minimal computational overhead, and is compatible with arbitrary base optimizers. Moreover, Gradient Smoothing admits a natural interpretation as a structured preconditioning method acting on block-structured parameter spaces, while providing a foundation for exploring richer forms of depth-wise update augmentation. Contributions. 1. We introduce Depth-wise Gradient Augmentation , a general optimization paradigm in which the collection of block-wise optimizer updates is transformed by a depth-wise augmentation operator before being applied. Within this framework, we study Gradient Smoothing , a family of structured depth-wise smoothing methods, and instantiate it with a simple, optimizer-agnostic Window Smoothing operator, providing a foundation for exploring more general forms of depth-wise update augmentation. 2. We demonstrate that Gradient Smoothing consistently improves optimization and generalization across a diverse set of architectures and training regimes, including language model pretraining, RL fine-tuning of LLMs for reasoning, supervised learning with Vision Transformers, and diffusion models. 3. We provide both empirical and theoretical evidence that Gradient Smoothing promotes more structured representation evolution across depth. Empirically, we observe increased layer-wise trajectory alignment and representation residual similarity, while theoretically we characterize how smoothing influences representational residual structure. 2 Background and Setup 2.1 Block-Structured Optimization We consider neural networks composed of L repeated blocks connected by skip connections, where each block shares the same (or similar) functional form but has its own parameters. In addition, the model may include a (typically small) collection of parameters not associated with the repeated structure, such as input embeddings or output heads. Formally, let F:\mathbb{R}^{d}\times\mathbb{R}^{p}\to\mathbb{R}^{d} denote the residual mapping associated with a single block. For block \ell , the residual function is given by F(\cdot;\theta_{\ell}) , where \theta_{\ell}\in\mathbb{R}^{p} are layer-specific parameters. Given the initial embedding h_{0} , the hidden states (h_{\ell})_{\ell=1}^{L}\subset\mathbb{R}^{d} then evolve according to h_{\ell+1}=h_{\ell}+F(h_{\ell};\theta_{\ell}),\qquad\ell=1,\dots,L-1. (1) Although we present the formulation for standard residual connections, the framework naturally extends to architectures with more general skip connection patterns. We write the full parameter vector as (\theta,\phi),\qquad\theta=(\theta_{1},\dots,\theta_{L}),\quad\theta_{l}\in\mathbb{R}^{p}, where \phi\in\mathbb{R}^{D_{\phi}} collects all non-block parameters. This setting encompasses a broad class of modern architectures, including residual networks with repeated block structure, Vision Transformers, and Transformer-based language models (by letting each h_{\ell} be a sequence of vectors). We denote the training objective by \mathcal{L}(\theta,\phi) and assume it is differentiable. At optimization step t , the block-wise gradients are given by g_{l}^{(t)}:=\nabla_{\theta_{l}}\mathcal{L}(\theta^{(t)},\phi^{(t)}),\qquad g^{(t)}:=(g_{1}^{(t)},\dots,g_{L}^{(t)}), and we denote by g_{\phi}^{(t)}:=\nabla_{\phi}\mathcal{L}(\theta^{(t)},\phi^{(t)}) the gradient with respect to \phi . Standard first-order methods treat the block-wise gradients \{g_{l}^{(t)}\}_{l=1}^{L} independently. However, repeated-block architectures induce strong structural relationships across depth: adjacent blocks often learn similar transformations and exhibit aligned representations, particularly after an initial transient phase of training. This suggests that the stacked gradient vector g^{(t)} possesses meaningful structure along the depth dimension. f_{L} f_{3} f_{2} f_{1} \vdots \mathcal{U}\!\left(\nabla_{\theta_{L}}L\right) \mathcal{U}\!\left(\nabla_{\theta_{3}}L\right) \mathcal{U}\!\left(\nabla_{\theta_{2}}L\right) \mathcal{U}\!\left(\nabla_{\theta_{1}}L\right) \vdots Filter S \tilde{\mathcal{U}}\!\left(\nabla_{\theta_{L}}L\right) \tilde{\mathcal{U}}\!\left(\nabla_{\theta_{3}}L\right) \tilde{\mathcal{U}}\!\left(\nabla_{\theta_{2}}L\right) \tilde{\mathcal{U}}\!\left(\nabla_{\theta_{1}}L\right) \vdots Figure 1 : Gradient Smoothing. Representation of the gradient augmentation scheme applied across depth to the updates in a deep network after backpropagation. For a deep network with L identical architectural blocks (but with different parameters), the gradient updates in each parameter block \theta^{l} are reweighted across depth to stabilize information propagation. (a) Depth 24 : Validation Loss. (b) Depth 24 : Validation BPB. (c) Depth 24 : CORE Metric. (d) Depth 30 : Validation Loss. (e) Depth 30 : Validation BPB. (f) Depth 30 : CORE Metric. Figure 2 : Nanochat pretraining with Gradient Smoothing. Validation loss, BPB, and DCLM CORE metric during nanochat pretraining of GPT2 with the default Adam + NorMuon optimizer setup (Table 1 ). We compare the baseline against a run with identical hyperparameters using depth-wise update smoothing applied to the transformer blocks. Models of depth 24 (1.38B parameters, top row) and depth 30 (2.40B parameters bottom row) with \alpha=0.1 are replicated against the baseline for \textbf{seed}\in\{42,67,2026\} . At both depths, smoothing accelerates convergence of validation loss and improves the CORE metric during training, with a larger gap visible at depth 30 . 2.2 General First-Order Updates Rather than committing to