PowerStep: Memory-Efficient Adaptive Optimization via ℓ p \ell_{p} -Norm Steepest Descent
Yao Lu 1 Dengdong Fan 1 Shixun Zhang 1 Yonghong Tian 1,2 Pengcheng Laboratory 1 Peking University 2 · 2026
Read PowerStep: Memory-Efficient Adaptive Optimization via ℓ p \ell_{p} -Norm Steepest Descent by Yao Lu 1 Dengdong Fan 1 Shixun Zhang 1 Yonghong Tian 1,2 Pengcheng Laboratory 1 Peking University 2 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10335.txt.
Introduction
Yao Lu and three colleagues cut the memory cost of training giant neural networks. Adaptive trainers such as Adam dominate large-scale training, but they store running estimates of gradient first and second moments, doubling trainer memory. Their 2026 paper introduces PowerStep, a trainer that achieves per-coordinate adaptivity with no second-moment storage.
Motivated by steepest descent under an Lp-norm geometry, PowerStep applies a nonlinear transform straight to a momentum buffer, yielding per-coordinate adaptivity from one buffer alone. The authors prove convergence at the optimal one-over-square-root-of-T rate for non-convex stochastic training. On Transformers from 124 million to 235 billion parameters, PowerStep matched Adam's speed while halving trainer memory. With aggressive int8 quantization it stayed stable and cut trainer memory about 8 times against full-precision Adam.
Catalog record
| CREATOR | Yao Lu 1 Dengdong Fan 1 Shixun Zhang 1 Yonghong Tian 1,2 Pengcheng Laboratory 1 Peking University 2 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.10335
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PowerStep: Memory-Efficient Adaptive Optimization via ℓ p \ell_{p} -Norm Steepest Descent Yao Lu 1 Dengdong Fan 1 Shixun Zhang 1 Yonghong Tian 1,2 Pengcheng Laboratory 1 Peking University 2 Abstract Adaptive optimizers, most notably Adam, have become the default standard for training large-scale neural networks such as Transformers. These methods maintain running estimates of gradient first and second moments, incurring substantial memory overhead. We introduce PowerStep, a memory-efficient optimizer that achieves coordinate-wise adaptivity without storing second-moment statistics. Motivated by steepest descent under an ℓ p \ell_{p} -norm geometry, we show that applying a nonlinear transform directly to a momentum buffer yields coordinate-wise adaptivity. We prove that PowerStep converges at the optimal O ( 1 / T ) O(1/\sqrt{T}) rate for non-convex stochastic optimization. Extensive experiments on Transformer models ranging from 124M to 235B parameters demonstrate that PowerStep matches Adam’s convergence speed while halving optimizer memory. Furthermore, when combined with aggressive int8 quantization, PowerStep remains numerically stable and reduces optimizer memory by ∼ 8 × \sim\!8\times compared to full-precision Adam. PowerStep thus provides a principled, scalable and resource-efficient alternative for large-scale training. Code is available at https://github.com/yaolubrain/PowerStep . 1 Introduction Neural networks are notoriously difficult to train, owing in large part to the vanishing and exploding gradient problem [ 21 ; 4 ] and the ill-conditioned curvature of the loss landscape [ 26 ; 11 ; 10 ] , which can cause stochastic gradient descent (SGD) to converge slowly or even diverge. Adaptive gradient methods [ 15 ; 40 ; 24 ; 29 ] alleviate these issues by preconditioning gradient updates with accumulated second-moment statistics. Among these methods, Adam [ 24 ] and its decoupled weight decay variant AdamW [ 29 ] have become the default optimizers for training large-scale neural networks, particularly Transformers [ 43 ] . Consider the optimization of an objective function f(\bm{\theta}):\mathbb{R}^{d}\to\mathbb{R} . Let \mathbf{g}_{t} denote the stochastic gradient at step t , while \mathbf{m}_{t} and \mathbf{v}_{t} represent the exponential moving averages (EMA) of the first and second moments of \mathbf{g}_{t} , respectively. The update rule for Adam [ 24 ] is defined as \bm{\theta}_{t}=\bm{\theta}_{t-1}-\eta_{t}\cdot\mathbf{m}_{t}/(\sqrt{\mathbf{v}_{t}}+\epsilon), (1) where \eta_{t} is the learning rate and \epsilon is a small constant. Intuitively, \mathbf{v}_{t} acts as a coordinate-wise scaling factor: by dividing the first moment by the square root of the second moment, Adam dampens parameters with large historical gradients and amplifies those with small ones. Equivalently, this applies a diagonal preconditioner \mathbf{D}_{t}=\mathrm{diag}((\sqrt{\mathbf{v}_{t}}+\epsilon)^{-1}) to the momentum direction. This adaptivity enables Adam to converge substantially faster than SGD, a phenomenon whose underlying mechanisms are still under active investigation [ 2 ; 33 ; 25 ; 48 ; 41 ; 51 ; 22 ] . However, the reliance on the second-moment accumulator \mathbf{v}_{t} incurs a significant memory overhead. Since \mathbf{v}_{t} shares the same dimensionality as the model parameters \bm{\theta}_{t} and typically requires fp32 precision to maintain numerical stability, it doubles the optimizer state footprint relative to SGD with momentum. This “memory wall” has motivated a growing body of work aimed at reducing optimizer memory. One line of work approximates the second-moment matrix through factorization [ 38 ] or blockwise sharing [ 52 ; 49 ] . Another applies low-precision quantization to the optimizer states [ 14 ; 27 ; 20 ] . Alternatively, several optimizers dispense with the second moment buffer entirely [ 6 ; 2 ; 8 ; 47 ] . Orthogonal to the adaptive optimization methods, a parallel line of work explores optimization under an \ell_{p} -norm geometry. Early work by Grove et al. [ 19 ] and Gentile [ 16 ] established convergence guarantees for \ell_{p} -norm algorithms in linear classification and regression. More recently, this geometric perspective has been adapted to accelerate gradient-based optimization in broader model classes. The Powerball method [ 46 ] leverages \ell_{p} -norm steepest descent to update parameters directly, while pbSGDM [ 53 ] integrates this geometry with momentum accumulation by applying a signed power transform before momentum. Stacey [ 31 ] further extends the approach, applying the transform to a momentum buffer within a primal-dual interpolation framework to accelerate SGD in non-convex settings. In this paper, we establish a direct connection between adaptive optimization and \ell_{p} -norm steepest descent. Our contributions are as follows: • Algorithm design. We derive PowerStep from \ell_{p} -norm steepest descent principles and show that applying a signed power transform directly to a heavy-ball momentum buffer yields coordinate-wise adaptivity without storing any second-moment statistics. • Theoretical analysis. We establish an optimal O(1/\sqrt{T}) convergence rate for PowerStep in non-convex stochastic optimization. • Empirical validation. We evaluate PowerStep on Transformer models ranging from 124M to 235B parameters, spanning both dense and mixture-of-experts (MoE) architectures. PowerStep matches the convergence speed of AdamW while using only half the optimizer state memory. Furthermore, PowerStep remains numerically stable under aggressive int8 quantization, reducing optimizer memory by \sim\!8\times compared to full-precision AdamW. 2 Algorithm In this section, we present PowerStep, a memory-efficient optimizer that achieves adaptive convergence without second-moment estimation. We derive its update rule from first principles and provide a geometric interpretation of its behavior. 2.1 Steepest descent on \ell_{p} -norm geometry The goal of first-order optimization is to identify an update direction \mathbf{v} that maximally decreases the objective f(\bm{\theta}) . In the framework of steepest descent in general normed spaces [ 7 ] , the optimal direction is defined as \mathbf{v}^{*}=\arg\min_{\mathbf{v}\in\mathbb{R}^{d}}\langle\nabla f(\bm{\theta}),\mathbf{v}\rangle\quad\text{s.t. }\|\mathbf{v}\|\leq 1. (2) The geometry of the trust region is dictated by the choice of norm. By adopting the Minkowski \ell_{p} norm, \|\mathbf{v}\|_{p}=(\sum_{i=1}^{d}|v_{i}|^{p})^{1/p} , we define a continuum of geometries. To derive the analytical update, we form the Lagrangian using the p -th power of the norm constraint \mathcal{L}(\mathbf{v},\mu)=\sum_{i=1}^{d}g_{i}v_{i}+\mu\left(\sum_{i=1}^{d}|v_{i}|^{p}-1\right), (3) where g_{i} is the i -th component of the gradient \mathbf{g}=\nabla f(\bm{\theta}) and \mu>0 is the Lagrange multiplier. Setting \frac{\partial\mathcal{L}}{\partial v_{i}}=0 yields the coordinate-wise update rule \displaystyle v_{i}^{*}\propto-\operatorname{sign}(g_{i})\cdot|g_{i}|^{\frac{1}{p-1}}. (4) Updating the parameters directly along this direction in a Euclidean manifold leads to \ell_{p} -norm steepest descent, also known as the Powerball method [ 46 ] . 2.2 Adaptivity via \ell_{p} -norm steepest descent Motivated by steepest descent direction \mathbf{v}^{*} in ( 4 ), we define the signed power transform as \displaystyle\Phi_{\beta}(\mathbf{x})=\operatorname{sign}(\mathbf{x})\odot|\mathbf{x}|^{\beta}, (5) where \beta=1/(p-1)\in[0,1] , \odot denotes elementwise multiplication and all other operations ( \operatorname{sign}(\cdot) , |\cdot| and (\cdot)^{\beta} ) are applied elementwise. As illustrated in Figure 1 , for \beta<1 , the transform nonlinearly dampens large magnitudes while amplifying small ones. Figure 1: Signed power transform \Phi_{\beta}(x) for different \beta values We propose PowerStep, which applies this nonlinear transform to the heavy-ball momentum [ 34 ] . Unlike Powerball that transforms raw gradients, PowerStep accumulates gradients linearly and applies \Phi_{\beta} after momentum integration, allowing temporal smoothing of the noisy gradients, \displaystyle\mathbf{m}_{t} \displaystyle=\gamma\mathbf{m}_{t-1}+\mathbf{g}_{t}, (6) \displaystyle\bm{\theta}_{t} \displaystyle=\bm{\theta}_{t-1}-\eta_{t}\Phi_{\beta}(\mathbf{m}_{t}). (7) The complete procedure is stated in Algorithm 1 . Equivalently, the PowerStep update ( 7 ) can be expressed as a preconditioned momentum step, \displaystyle\bm{\theta}_{t} \displaystyle=\bm{\theta}_{t-1}-\eta_{t}\mathbf{D}_{t}\mathbf{m}_{t}, (8) \displaystyle\mathbf{D}_{t} \displaystyle=\operatorname{diag}\bigl(|\mathbf{m}_{t}|^{\beta-1}\bigr), (9) where \mathbf{D}_{t} is a diagonal preconditioner whose i -th entry is |m_{t,i}|^{\beta-1} , with m_{t,i} denoting the i -th component of \mathbf{m}_{t} . This formulation reveals the source of PowerStep’s adaptivity. For \beta\in(0,1) , the exponent \beta-1 is strictly negative, so each |m_{t,i}|^{\beta-1} is inversely related to the local momentum magnitude: when |m_{t,i}| is small