Outer-Momentum Restarting in High-Dimensional Two-Phase Optimization

Kristi Topollai, Allan Ma, Tolga Dimlioglu, Sui Jiet Tay, Anna Choromanska · 2026

Read Outer-Momentum Restarting in High-Dimensional Two-Phase Optimization by Kristi Topollai, Allan Ma, Tolga Dimlioglu, Sui Jiet Tay, Anna Choromanska (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-28585.txt.

Introduction

Kristi Topollai and four colleagues study periodic restarting of outer momentum in low-talk distributed tuning. Methods like DiLoCo let workers run many local updates before merging them with an outer momentum optimizer, and outer momentum shapes how local progress adds up. In a linearized squared-loss model, they derive a mode-wise restart contraction: resets exploit phase cancellation by dropping stale momentum while keeping inner-loop progress. Toy tests confirm the contraction, and language-model pretraining shows restarts broaden the stable band of outer learning rates and momentum settings.

Catalog record

CREATORKristi Topollai, Allan Ma, Tolga Dimlioglu, Sui Jiet Tay, Anna Choromanska
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38614
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.28585

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Outer-Momentum Restarting in High-Dimensional Two-Phase Optimization Kristi Topollai, Allan Ma, Tolga Dimlioglu, Sui Jiet Tay, Anna Choromanska Abstract Communication-efficient distributed optimizers such as DiLoCo reduce synchronization costs by letting workers perform many local updates before aggregating their progress with an outer momentum optimizer. Recent theory suggests that the outer optimizer acts on an effective spectrum induced by the inner optimization loop, and that the choice of outer momentum controls how progress from local updates is accumulated across communication rounds. We study periodic restarting of the outer momentum as a simple complementary mechanism for controlling this outer memory. In a linearized squared-loss model where prediction-space residuals evolve under the empirical NTK, we derive a mode-wise restart contraction showing that resets exploit phase cancellation by discarding stale momentum while preserving inner-loop progress. Toy experiments verify the predicted contraction behavior, and language-model pretraining experiments show that periodic restarts widen the stable range of outer learning rates and momentum values across communication periods. \hldauthor New York University, New York, USA 1 Introduction Modern language-model (LM) training is increasingly constrained by communication in distributed setups. In standard synchronous data-parallel training, workers exchange gradients or optimizer states at every step, which becomes costly when accelerators such as GPUs or TPUs are distributed across machines or clusters. This led to renewed interest in local and two-phase training methods, where workers perform several local optimization steps before synchronizing 19 ; 18 ; 21 ; 3 ; 4 ; 15 . In this view, training is split into an inner phase, which performs independent local progress, and an outer phase, which aggregates that progress across workers. Similar to Lookahead methods 26 , DiLoCo brings this structure to large-scale LM training: workers run many inner steps, form a pseudo-gradient from their parameter displacement, and pass it to an outer optimizer 8 ; 7 . Empirically, DiLoCo and its variants can substantially reduce communication while maintaining training quality, and recent work suggests that some benefits arise even in the one-worker setting 8 ; 14 . This way, the outer optimizer determines how accumulated local progress is filtered, or damped at communication time. Recent works on local SGD and federated optimization study how outer learning rates and momentum trade off optimization speed against stochastic noise 21 ; 4 ; 15 . The closest theoretical lens is provided in 1 , where it is shown that the inner loop induces an effective spectrum on which the outer optimizer operates. Their analysis shows that heavy-ball outer momentum can constrain directions on which the inner loop already makes strong progress, while Nesterov outer momentum mitigates this by making contraction depend on the effective progress of each mode 1 . This is consistent with recent empirical gains from Nesterov pseudo-gradient updates 14 . We study momentum restarting as a complementary mechanism for two-phase optimizers. Rest-arting has a long history in accelerated optimization, where resetting momentum can reduce oscillation or correct misalignment 20 ; 10 ; 9 ; 16 , with related restart ideas being used in deep learning to improve stability 24 ; 12 ; 23 . In two-phase training, the outer momentum buffer stores pseudo-gradient history only at communication rounds, which means that outdated outer memory can affect how local progress is aggregated across synchronizations. We show that periodically resetting this buffer reduces fragility to the outer momentum and learning rate. Our theory explains the effect through a phase-dependent contraction, while our 150 M-parameter LLaMA pretraining experiments show that restarts widen the stable region of outer-optimizer hyperparameters, a useful property in pretraining where optimizer tuning is already delicate 25 ; 5 and two-phase methods introduce an additional outer-loop hyperparameter layer. 2 Preliminaries: deterministic two-phase mode dynamics We introduce a deterministic mode-wise model to isolate how the outer optimizer processes the progress accumulated by the inner loop. To do so, we follow the two-phase analysis of 1 . The object being analyzed is the training residual in prediction space, R(\theta)=f(\theta)-y_{\mathrm{tr}} where f(\theta) denotes the vector of model predictions on the training set and y_{\mathrm{tr}} denotes the training targets. Under the standard linearized squared-loss/NTK approximation, the residual dynamics are governed by the empirical kernel H . After diagonalizing this kernel, the residual decomposes into independent scalar modes, so we use a deterministic scalar-mode surrogate for the residual dynamics rather than modeling the full nonlinear parameter trajectory 13 ; 17 ; 2 . We analyze one residual eigencoordinate at a time. Let \lambda\geq 0 denote the corresponding empirical-kernel eigenvalue. Within one communication round, the inner optimizer performs S gradient-descent steps with step size \eta where 0\leq\eta\lambda\leq 1 . If x_{t} is the value of this residual mode at the beginning of outer round t , then after the inner steps, x_{t}^{\mathrm{loc}}=(1-\eta\lambda)^{S}x_{t},\qquad g_{t}:=x_{t}-x_{t}^{\mathrm{loc}}=\sigma x_{t},\qquad\sigma:=1-(1-\eta\lambda)^{S}.\vskip-2.84526pt (1) Here x_{t}^{\mathrm{loc}} denotes the value of the same residual mode after the worker has completed S local inner steps and g_{t} is the pseudo-gradient passed to the outer optimizer. The coefficient \sigma\in[0,1] is the effective progress seen by the outer optimizer at communication time and large \sigma means that the inner loop has already made substantial progress on that residual mode. For heavy-ball/EMA outer momentum, let m_{t} denote the outer momentum buffer. At outer round t , the pseudo-gradient g_{t} is accumulated through m_{t+1}=\beta_{\mathrm{out}}m_{t}+(1-\beta_{\mathrm{out}})g_{t} , and the residual eigencoordinate is updated as x_{t+1}=x_{t}-\nu m_{t+1} . Since the inner loop gives g_{t}=\sigma x_{t} , the pair consisting of the residual mode and its outer momentum, z_{t}=(x_{t},m_{t})^{\top} , evolves linearly across communication rounds. Substituting g_{t}=\sigma x_{t} into the two scalar updates yields z_{t+1}=T_{\mathrm{HB}}(\sigma)z_{t}\qquad\text{where}\qquad T_{\mathrm{HB}}(\sigma)=\begin{pmatrix}1-\nu(1-\beta_{\mathrm{out}})\sigma&-\nu\beta_{\mathrm{out}}\\ (1-\beta_{\mathrm{out}})\sigma&\beta_{\mathrm{out}}\end{pmatrix}.\vskip-2.84526pt (2) Since \det(T_{\mathrm{HB}})=\beta_{\mathrm{out}} , complex-regime eigenvalues have magnitude \sqrt{\beta_{\mathrm{out}}} , giving the \sigma -indepen-dent damping rate r_{\infty}=-\frac{1}{2}\log\beta_{\mathrm{out}} . This is the outer-momentum damping effect, that is, even high-progress modes with \sigma\approx 1 can be limited by the same slow HB envelope. For Nesterov-style outer momentum, the buffer update is unchanged, but the outer step uses x_{t+1}=x_{t}-\nu((1+\beta_{\mathrm{out}})m_{t+1}-\beta_{\mathrm{out}}m_{t}) , which gives T_{\mathrm{NAG}}(\sigma)=\begin{pmatrix}1-\nu(1-\beta_{\mathrm{out}}^{2})\sigma&-\nu\beta_{\mathrm{out}}^{2}\\ (1-\beta_{\mathrm{out}})\sigma&\beta_{\mathrm{out}}\end{pmatrix},\qquad\det(T_{\mathrm{NAG}})=\beta_{\mathrm{out}}\bigl(1-(1-\beta_{\mathrm{out}})\nu\sigma\bigr). (3) 3 Restarting Outer Momentum: An Old Trick Revisited The preceding discussion explains why Nesterov outer momentum is effective in two-phase training. For HB, complex modes share the same envelope \rho=\sqrt{\beta_{\mathrm{out}}} , so the outer update may remain slow even when the inner loop has made substantial progress. NAG addresses this spectrally as its contraction depends on \nu\sigma , making the outer optimizer more responsive to high-progress modes. We study a complementary mechanism: periodic restarting of the outer momentum buffer. Momentum restarting is classical in accelerated optimization, where resets can reduce oscillations or correct momentum misalignment 20 ; 10 ; 9 ; 16 ; 22 . In our setting, after every K outer rounds, we reset only the outer buffer, m\leftarrow 0 . Unlike NAG, restarting does not modify the one-step spectrum; it modifies the duration over which outer memory is retained. In the complex regime, the residual coordinate oscillates within an envelope, so an appropriately chosen restart period can stop the trajectory near a phase cancellation, where the projected residual is much smaller than the envelope. Thus NAG and restarting act through distinct mechanisms: NAG changes the contraction factor, whereas restarting exploits favorable reset times. Starting a restart cycle with zero momentum, m\leftarrow 0 , one cycle of K outer rounds maps the initial residual mode to x_{K}=\chi_{K}(\sigma)x_{0},\qquad\chi_{K}(\sigma):=\bigl[T_{\mathrm{HB}}(\sigma)^{K}\bigr]_{11},\qquad r_{K}(\sigma):=-\frac{1}{K}\log|\chi_{K}(\sigma)|.

Read the complete text