Worker Disagreement Reveals Sharp Directions in Local SGD

Tolga Dimlioglu, Kristi Topollai, Anna Choromanska · 2026

Read Worker Disagreement Reveals Sharp Directions in Local SGD by Tolga Dimlioglu, Kristi Topollai, Anna Choromanska (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27739.txt.

Introduction

Tolga Dimlioglu, Kristi Topollai, and Anna Choromanska show that worker disagreement in Local SGD exposes sharp directions in loss shape. The worker-average gap covariance reflects gradient noise and Hessian curvature, so gaps between workers trace bend-sensitive sharp directions. Worker-average gaps therefore give a cheap Hessian-free gauge of the dominant subspace, catching much of the gradient in that eigenspace.

Catalog record

CREATORTolga Dimlioglu, Kristi Topollai, Anna Choromanska
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54663
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.27739

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Worker Disagreement Reveals Sharp Directions in Local SGD Tolga Dimlioglu, Kristi Topollai, Anna Choromanska Abstract Deep neural network training often exhibits highly anisotropic loss geometry, where a few sharp dominant Hessian directions coexist with a large flatter bulk. Gradients tend to align disproportionately with these dominant directions, although stable progress often requires movement through flatter bulk directions 13 . Estimating the dominant subspace is therefore useful but costly with direct Hessian-based methods. We show that standard Local SGD exposes this geometry through worker disagreement. We theoretically show that the worker–average gap covariance is shaped by stochastic-gradient noise and Hessian curvature, causing workers to disagree along sharp, curvature-sensitive directions. Thus, worker–average gaps provide a cheap Hessian-free estimator of the dominant subspace. Experiments on MLPs, CNNs, and Transformers show that subspaces formed by worker–average gaps capture a substantial fraction of the gradient component lying in the dominant Hessian eigenspace. \hldauthor New York University, New York, USA 1 Introduction Deep neural networks are optimized in extremely high-dimensional parameter spaces, yet their training dynamics often exhibit low-dimensional structure 7 ; 17 . In particular, the training-loss Hessian is highly anisotropic: a few large eigenvalues define a sharp low-dimensional dominant subspace, while the remaining directions form a flatter high-dimensional bulk ( 10 ; 11 ; 9 ; 2 ) . Gradients along centralized SGD trajectories often concentrate in the dominant eigenspace ( 3 ; 1 ) , but recent evidence shows that useful loss descent can depend substantially on movement through flatter bulk directions ( 13 ) . Thus, identifying the dominant subspace is important for diagnosing and controlling optimization, but direct Hessian-based estimation is computationally expensive. We investigate whether distributed training dynamics can reveal the dominant subspace without explicitly estimating the Hessian. In Local SGD, workers independently take multiple stochastic-gradient steps before averaging ( 14 ; 8 ) , naturally disagreeing on the next iterate. We quantify this disagreement via worker–average gaps. Prior analyses typically treat these gaps as consensus error to be bounded ( 14 ; 4 ) . We instead ask whether they reveal useful landscape directions. We answer this question theoretically and empirically. We show that worker–average gap covariance is shaped by the interaction between stochastic-gradient noise and Hessian curvature, amplifying disagreement along directions that are both noisy and curvature-sensitive. Since SGD noise is highly anisotropic and curvature-aligned ( 20 ; 16 ; 15 ; 18 ) , worker–average gaps can serve as a Hessian-free proxy for the dominant eigenspace. Experiments on MLPs, CNNs, and Transformers show that worker–average gap subspaces capture, and can be used to suppress, a substantial fraction of the gradient’s dominant Hessian component. These results recast worker disagreement from a consensus error into a low-cost source of geometric tool for controlling optimization. 2 Preliminaries and Initial Observations Setup. We consider distributed empirical risk minimization with M workers minimizing f(\theta)=\frac{1}{N}\sum_{n=1}^{N}\ell(h_{\theta}(x_{n}),y_{n}) , where h_{\theta}:\mathcal{X}\to\mathcal{Y} is a neural network with parameters \theta\in\mathbb{R}^{D} and \ell is the per-sample loss. The training set is partitioned IID across M workers. We focus on Local SGD training. At the beginning of each communication round, all workers start from a common parameter vector; each worker then performs \tau local stochastic-gradient steps on its own minibatches; and the workers synchronize by averaging their parameters. Pseudo-code is provided in Appendix 1 . Notations. Here, we follow the dominant–bulk subspace setup of ( 13 ) . Let \nabla f(\theta)\in\mathbb{R}^{D} denote the full-batch gradient and H(\theta)=\nabla^{2}f(\theta)\in\mathbb{R}^{D\times D} denote the loss Hessian. Let \lambda_{1}(\theta)\geq\cdots\geq\lambda_{D}(\theta) be the eigenvalues of H(\theta) , with corresponding eigenvectors u_{1}(\theta),\ldots,u_{D}(\theta) . For a chosen dimension C , we define the Dominant subspace as \mathcal{S}_{C}(\theta)=\operatorname{span}\{u_{1}(\theta),\ldots,u_{C}(\theta)\} , and refer to its orthogonal complement \mathcal{S}_{C}^{\perp}(\theta) as the Bulk subspace. Following ( 3 ; 13 ) , we set C to the number of classes in the classification task. We denote the orthogonal projector onto \mathcal{S}_{C}(\theta) by P_{C}(\theta)=\sum_{k=1}^{C}u_{k}(\theta)u_{k}(\theta)^{\top} , and define P_{C}^{\perp}(\theta)=I-P_{C}(\theta) . For any vector v\in\mathbb{R}^{D} , we measure its relative alignment with the dominant subspace using \chi_{C}(v;\theta):=\|P_{C}(\theta)v\|_{2}/\|v\|_{2}\vskip-3.98337pt (1) This metric is inherited from 13 . Notice that, when \chi_{C}(v;\theta) is close to one, we say v is aligned with \mathcal{S}_{C}(\theta) as most of the norm of v lies in the dominant subspace; when it is close to zero, v lies mostly in the bulk subspace. Next, we verify that the dominant-subspace alignment observed in centralized SGD also appears along Local SGD trajectories. Figure 1 : FC tanh network trained on MNIST-5k using Local SGD with M=4 workers and \tau=5 . Left: the Hessian spectrum develops a clear separation between the top 10 eigenvalues and the bulk. Middle: the gradient direction increasingly aligns with the dominant Hessian subspace, measured by \chi_{10} . Right: retaining only the dominant component stalls optimization, indicating that useful descent is largely carried by bulk directions. Observation 1: Local SGD gradients align with the dominant subspace. At each synchronization round, after worker averaging, we compute the full-batch gradient \nabla f(\bar{\theta}) and Hessian H(\bar{\theta}) at the averaged parameter \bar{\theta} , form the dominant subspace \mathcal{S}_{C}(\bar{\theta}) , and report \chi_{C}(\nabla f(\bar{\theta});\bar{\theta}) . Figure 1 shows the result for a fully-connected (FC) tanh network trained on MNIST 6 with M=4 workers and communication period \tau=5 . The Hessian develops a clearly separated dominant subspace, and the gradient increasingly aligns with this subspace throughout training as shown in left and middle panels of Figure 1 respectively. We observe the same qualitative behavior for a CNN on CIFAR10 5 and a Transformer on SST2 12 ; but these results are deferred to Appendix A.2 due to space constraints. Then, we test whether this dominant-space alignment implies that Local SGD can be continued using only the dominant component of its communication-round update. Observation 2: Local SGD progress is not carried by the dominant component. Starting from a standard Local SGD checkpoint, we compare standard Local SGD with two projected variants. Let \bar{p}^{c}:=\frac{1}{M}\sum_{i=1}^{M}\theta_{i}^{c,\tau}-\bar{\theta}^{c} denote the average outer step at communication round c . Dom-Local SGD projects \bar{p}^{c} onto the dominant subspace \mathcal{S}_{C}(\bar{\theta}^{c}) , whereas Bulk-Local SGD projects it onto the complementary bulk subspace \mathcal{S}_{C}^{\perp}(\bar{\theta}^{c}) . As shown in Figure 1 right panel, Dom-Local SGD stalls after the projection is introduced, whereas Bulk-Local SGD closely tracks standard Local SGD. Thus, although gradients are strongly aligned with the dominant Hessian subspace, the update component that sustains training lies primarily in the bulk. More details and results on CIFAR-10 and SST-2 are given in Appendix A.2 . Next, we look beyond the average outer step of Local SGD and analyze the disagreement among local worker parameters, i.e., the gap vector between each worker parameter and their average, which Local SGD produces naturally during training. 3 Theoretical Characterization of Worker–Average Gaps We analyze how worker disagreements, i.e., the worker–average gaps, arise within communication round c . At the beginning of the round, all workers are synchronized at \bar{\theta}^{c} , so that \theta_{i}^{c,0}=\bar{\theta}^{c} for i=1,\ldots,M . Each worker then performs \tau local SGD steps with step size \eta : \theta_{i}^{c,t}=\theta_{i}^{c,t-1}-\eta g_{i}^{c,t},\qquad g_{i}^{c,t}=\nabla f(\theta_{i}^{c,t-1})+\epsilon_{i}^{c,t},\qquad t=1,\ldots,\tau, (2) where \epsilon_{i}^{c,t} is the stochastic-gradient noise satisfying \mathbb{E}[\epsilon_{i}^{c,t}\mid\theta_{i}^{c,t-1}]=0 . Now, define the within-round average and the worker–average gap i from this average as \bar{\theta}^{c,t}:=\frac{1}{M}\sum_{j=1}^{M}\theta_{j}^{c,t} and d_{i}^{c,t}:=\theta_{i}^{c,t}-\bar{\theta}^{c,t} respectively. Notice that, by construction, \sum_{i}d_{i}^{c,t}=0 , and since workers are synchronized at the start of

Read the complete text