Generalized Gradient Norm Clipping & Non-Euclidean ( L 0 , L 1 ) (L_{0},L_{1}) -Smoothness
Thomas Pethick, Wanyun Xie, Mete Erdogan, Kimon Antonakopoulos, Antonio Silveti-Falls, Volkan Cevher · 2025
Read Generalized Gradient Norm Clipping & Non-Euclidean ( L 0 , L 1 ) (L_{0},L_{1}) -Smoothness by Thomas Pethick, Wanyun Xie, Mete Erdogan, Kimon Antonakopoulos, Antonio Silveti-Falls, Volkan Cevher (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01913.txt.
Introduction
Thomas Pethick and co-authors generalize gradient clipping, a trick used to keep model training stable. Clipping caps the size of an update step, and their 2025 paper blends it with two older ideas: steepest descent and conditional gradient steps. The blend works under a wider notion of smoothness that allows non-Euclidean geometry.
The team also folds weight decay into the method through a link to the Frank-Wolfe short step. With random data, their momentum-based version reaches an order-optimal convergence rate. They name the deep learning form Clipped Scion and test it on image classification and language modeling. Optimization readers get a clipping method with proofs, not just habits, behind it.
Catalog record
| CREATOR | Thomas Pethick, Wanyun Xie, Mete Erdogan, Kimon Antonakopoulos, Antonio Silveti-Falls, Volkan Cevher |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-01913
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generalized Gradient Norm Clipping & Non-Euclidean ( L 0 , L 1 ) (L_{0},L_{1}) -Smoothness Thomas Pethick, Wanyun Xie, Mete Erdogan, Kimon Antonakopoulos, Antonio Silveti-Falls, Volkan Cevher Abstract This work introduces a hybrid non-Euclidean optimization method which generalizes gradient norm clipping by combining steepest descent and conditional gradient approaches. The method achieves the best of both worlds by establishing a descent property under a generalized notion of ( L 0 L_{0} , L 1 L_{1} )-smoothness. Weight decay is incorporated in a principled manner by identifying a connection to the Frank-Wolfe short step. In the stochastic case, we show an order optimal O ( n − 1 / 4 ) O(n^{-1/4}) convergence rate by leveraging a momentum based gradient estimator. We discuss how to instantiate the algorithms for deep learning and demonstrate their properties on image classification and language modeling. 1 Introduction Recent work ( Pethick et al. 2025 ) has shown that conditional gradient methods 1 1 1 By conditional gradient based methods, we mean those methods which leverage a linear minimization oracle \operatorname{lmo}(d)=\argmin\limits_{x\in\mathcal{D}}\langle d,x\rangle when updating their parameters with an open-loop stepsize. , traditionally used for constrained optimization, can also solve unconstrained problems—offering an alternative to steepest descent. From their analysis it becomes apparently that the two methods have distinct properties: Whereas steepest descent requires the stepsize \gamma for L -smooth objectives to be taken as \gamma<\nicefrac{{2}}{{L}} , conditional gradient methods have no such requirement, thus allowing for large stepsizes, while remaining stable. The price to pay for the stability is that conditional gradient based methods are not descent methods and thus eventually needs a diminishing stepsize to converge, even in the deterministic case. The problem becomes very apparent if the iterates are close to the solution, since the iterates always move by a fixed magnitude and are thus pushed away from the solution. Steepest descent does not suffer from the same problem since the effective stepsize automatically becomes smaller as the iterates approach a solution. This observation naturally raises the following question: Can we combine the two methods and get the best of both worlds? That is, does a stable method exist which takes large steps initially but adapts the stepsize when near a solution? In this paper we answer the above in the affirmative by considering a hybrid method that combines a conditional gradient method with steepest descent. The proposed method generalizes gradient norm clipping ( Mikolov et al. 2012 ) beyond the Euclidean case. In practice, gradient norm clipping has been widely adopted to stabilize training of recurrent neural networks (RNNs), Transformers and diffusion models, especially in large-scale settings. Theoretically, a precise characterization of the benefits has emerged under the (L_{0},L_{1}) -smoothness assumption ( Zhang et al. 2019 ; Zhang et al. 2020 ; Koloskova et al. 2023 ) . Expanding on this, we show that these benefits of clipping can be made compatible with non-Euclidean methods. Besides clipping, we provide a novel analysis of conditional gradient methods without clipping under these same smoothness assumptions. Concretely, we make the following contributions: (i) We introduce a hybrid method between a conditional gradient method and steepest descent ( Algorithm 1 ), which in the Euclidean case recovers gradient norm clipping. The benefit of the hybrid method is made precise by showing a descent property under a generalized (L_{0},L_{1}) -smoothness condition. (ii) In the stochastic case we show an order optimal O(n^{-1/4}) rate by leveraging a momentum estimator. Convergence for a clipped algorithm with stochastic feedback appears to be new even in the Euclidean case. (iii) We establish a connection between clipping and the short step from the Frank-Wolfe literature, which similarly enjoys a descent property. The connection enables us to combine clipping with weight decay in a principled manner that maintains convergence guarantees. We propose a stochastic variant of the short step ( Algorithm 2 ) and establish a O(n^{-1/4}) rate. (iv) We explicitly instantiate the algorithms for deep learning through a product norm over layers ( and ) and demonstrate their properties through experiments on image classification and language modeling. Table 1: Special instantiations of Algorithm 1 according to different choices of norm. Control on the norm of the parameters is guaranteed by the constrained variant of the method ( Algorithm 2 ). Method Norm type Norm ball \operatorname{lmo}(d) \|d\|_{*} Reference Clipped GD Vector Euclidean \|\cdot\|_{2} -ball -\tfrac{d}{\|d\|_{2}} \|d\|_{2} ( Mikolov et al. 2012 ) Clipped Sign Vector Max-norm \|\cdot\|_{\infty} -ball -\operatorname{sign}(d) \|d\|_{1} This paper Clipped Spectral Matrix Spectral norm \|\cdot\|_{\mathcal{S}_{\infty}} -ball -UV^{\top} 1 -\operatorname{tr}(\operatorname{lmo}(d)^{\top}d) This paper ClippedScion ( and ) Product Max-norm ball over layers \{r_{l}\operatorname{lmo}_{\|\cdot\|_{\mathcal{W}_{l}}}(d_{l})\}_{l\in[D]} -\sum_{l}\braket{r_{l}\operatorname{lmo}(d_{l}),d_{l}} This paper 1 The reduced SVD is given as d=U\operatorname{diag}(\sigma)V^{\top} . 2 Preliminaries Given a continuously differentiable objective function f:\mathcal{X}\rightarrow\mathbb{R} , the classical gradient descent method (GD) can be written as \addcontentsline{lla}{section}{\numberline q:GD}x^{k+1}=x^{k}-\gamma\nabla f(x^{k}) (GD) with a stepsize \gamma>0 . The normalized gradient descent method is in comparison defined as follows \addcontentsline{lla}{section}{\numberline q:NGD}x^{k+1}=x^{k}-\gamma\left[\rho\tfrac{\nabla f(x^{k})}{\|\nabla f(x^{k})\|_{2}}\right] (Normalized GD) where \rho>0 is some additional radius parameter. A hybrid variant is much more popular in practice: \addcontentsline{lla}{section}{\numberline q:ClipGD}x^{k+1}=x^{k}-\gamma\min\{1,\tfrac{\rho}{\|\nabla f(x^{k})\|_{2}}\}\nabla f(x^{k}). (Clipped GD) The iterates are generated by Normalized GD if \|\nabla f(x^{k})\|_{2} is large, but reduce to GD when \|\nabla f(x^{k})\|_{2} is small enough. Observation I Our first observation is that both GD and Normalized GD can be generalized to the non-Euclidean case. Define the sharp-operator ( Nesterov 2012 ; Kelner et al. 2014 ) , d^{\sharp}\in\argmax_{x\in\mathcal{X}}\{\braket{d,x}-\tfrac{1}{2}\|x\|^{2}\}. Then, we can write the (possibly non-Euclidean) steepest descent method (SD) as follows \addcontentsline{lla}{section}{\numberline q:SD}x^{k+1}=x^{k}-\gamma[\nabla f(x^{k})]^{\sharp} (SD) Observe that we recover GD when choosing the Euclidean \ell_{2} norm. Generalizing Normalized GD to non-Euclidean norms is possible by noticing that the normalization can be written in terms of the linear minimization oracle ( \operatorname{lmo} ) \operatorname{lmo}(d)\in\argmin_{x\in\mathcal{D}}\braket{d,x} where the constraint is a (typically non-Euclidean) norm-ball \mathcal{D}:=\{x\mid\|x\|\leq 1\} . By choosing the \ell_{2} -norm ball, Normalized GD can be seen as an instance of the so-called unconstrained conditional gradient method (uCG) ( Pethick et al. 2025 ) , \addcontentsline{lla}{section}{\numberline q:uSCG}\begin{split}x^{k+1}&=x^{k}+\gamma\rho\operatorname{lmo}(\nabla f(x^{k})).\end{split} (uCG) Observation II Our second central observation is that uCG can in general be considered a normalized version of steepest descent. This relationship follows from noticing that the sharp operator and \operatorname{lmo} can be defined in terms of each other. Specifically, we have that \addcontentsline{lla}{section}{\numberline q:lmo:sharp}\operatorname{lmo}(d)=-\tfrac{d^{\sharp}}{\|d\|_{*}} (1) In the following section we use this observation to generalize Clipped GD to the non-Euclidean case. 3 Method We propose the generalized gradient norm clipping method (GGNC) \addcontentsline{lla}{section}{\numberline q:GGNC}\begin{split}x^{k+1}&=x^{k}-\gamma\tau_{k}[d^{k}]^{\sharp}\quad\text{with}\quad\tau_{k}:=\min\{1,\tfrac{\rho}{\|d^{k}\|_{*}}\}.\end{split} (GGNC) There is freedom in how to compute the dual norm \|d^{k}\|_{*} due to the following equivalence property for the sharp operator, \|s\|_{*}^{2}=\|s^{\sharp}\|^{2}=\braket{s,s^{\sharp}} . This form is useful, e.g., in the Euclidean case where the sharp-operator is readily available, since then [d^{k}]^{\sharp}=d^{k} . For norm choices where the \operatorname{lmo} is more naturally available we can equivalently write GGNC as \addcontentsline{lla}{section}{\numberline q:GGNC:lmo}\begin{split}x^{k+1}=x^{k}+\gamma\eta_{k}\operatorname{lmo}(d^{k})\quad\text{with}\quad\eta_{k}:=\min\{\rho,\|d^{k}\|_{*}\}.\end{split} We have that \|d^{k}\|_{*}=-\braket{d^{k},\operatorname{lmo}(d^{k})} due to the definition of the dual norm and the optimality of \operatorname{lmo}(d^{k}) . So, provided that \operatorname{lmo} has been computed, we can obtain \|d^{k}\|_{*} with very little overhead. From this rewriting we also see that \rho can also be interpreted as the radius of the norm-ball constraint over which we compute the \operatorname{lmo} . Stochastic case In the deterministic case we can simply take