When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study

Jun Yan 1 Weiquan Huang 2 Jiankai Zuo 3 [email protected] [email protected] [email protected] Yujian Mo 2 Xi Fang 4 Chengliang Wu 1 [email protected] [email protected] [email protected] Zeming Wei 5 [email protected] 1 IT College, Shanghai Ocean University, Shanghai, 201306 2 School of Computer Science and Technology, Tongji University, Shanghai, 201804 3 SEIE, Suzhou University of Science and Technology, Suzhou, 215009 4 DP Tech, Shanghai, 200030 5 School of Mathematical Sciences, Peking University, Beijing, 100871 · 2026

Read When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study by Jun Yan 1 Weiquan Huang 2 Jiankai Zuo 3 [email protected] [email protected] [email protected] Yujian Mo 2 Xi Fang 4 Chengliang Wu 1 [email protected] [email protected] [email protected] Zeming Wei 5 [email protected] 1 IT College, Shanghai Ocean University, Shanghai, 201306 2 School of Computer Science and Technology, Tongji University, Shanghai, 201804 3 SEIE, Suzhou University of Science and Technology, Suzhou, 215009 4 DP Tech, Shanghai, 200030 5 School of Mathematical Sciences, Peking University, Beijing, 100871 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26929.txt.

Introduction

Yan and six colleagues ask whether the Muon optimizer, which orthogonalizes matrix updates, can improve adversarial training under strong and varied threat models. Theory shows Muon caps spectral-norm growth in updates without shrinking learned weights. Across five architectures and three threat norms plus their union, Muon matches SGD on convolutional networks and clearly beats AdamW on both convolutional networks and vision transformers. Optimizer geometry, they conclude, is a security-relevant design choice.

Catalog record

CREATORJun Yan 1 Weiquan Huang 2 Jiankai Zuo 3 [email protected] [email protected] [email protected] Yujian Mo 2 Xi Fang 4 Chengliang Wu 1 [email protected] [email protected] [email protected] Zeming Wei 5 [email protected] 1 IT College, Shanghai Ocean University, Shanghai, 201306 2 School of Computer Science and Technology, Tongji University, Shanghai, 201804 3 SEIE, Suzhou University of Science and Technology, Suzhou, 215009 4 DP Tech, Shanghai, 200030 5 School of Mathematical Sciences, Peking University, Beijing, 100871
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75775
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26929

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Muon Optimizer Meets Adversarial Training: A Theoretical and Empirical Study Jun Yan 1 Weiquan Huang 2 Jiankai Zuo 3 [email protected] [email protected] [email protected] Yujian Mo 2 Xi Fang 4 Chengliang Wu 1 [email protected] [email protected] [email protected] Zeming Wei 5 [email protected] 1 IT College, Shanghai Ocean University, Shanghai, 201306 2 School of Computer Science and Technology, Tongji University, Shanghai, 201804 3 SEIE, Suzhou University of Science and Technology, Suzhou, 215009 4 DP Tech, Shanghai, 200030 5 School of Mathematical Sciences, Peking University, Beijing, 100871 Abstract Adversarial training (AT) remains one of the most reliable empirical defenses against adversarial attacks. Its robustness critically depends on how the underlying min-max objective is optimized. In practice, Stochastic Gradient Descent (SGD) optimizer remains the default optimization choice for AT, whereas adaptive optimizers often improve standard training but may yield inferior robustness. Recently, the Muon optimizer, which orthogonalizes matrix-valued updates via an approximate polar decomposition, has achieved notable success in large-scale training at a memory cost comparable to SGD. This raises a security-relevant question: can orthogonalized optimization improve AT under strong and heterogeneous threat models? Focusing on this problem, we conduct a comprehensive theoretical and empirical study. Theoretically, we show that Muon imposes a spectral-norm stability ceiling on matrix updates, limiting uncontrolled spectral growth in the training dynamics without explicitly shrinking the learned weights. Empirically, across five architectures and three ℓ p \ell_{p} threat models ( ℓ ∞ \ell_{\infty} , ℓ 1 \ell_{1} , ℓ 2 \ell_{2} ) and their union, Muon is competitive with SGD on CNNs and substantially outperforms AdamW on both CNNs and ViTs. These results identify optimizer geometry as a security-relevant factor in adversarial training, while clarifying the empirical regimes in which orthogonalized updates are beneficial. Overall, our findings highlight optimizer design as a security-critical component of AT. 1 Introduction Deep neural networks are increasingly deployed in security-sensitive settings, yet their predictions can be manipulated by small, human-imperceptible adversarial perturbations szegedy2014intriguing ; goodfellow2015explaining ; madry2018towards ; carlini2017towards . To mitigate such potential threats to neural networks, robust training methods known as adversarial training (AT) have been proposed madry2018towards ; zhang2019theoretically ; wong2020fast ; croce2022adversarial . These methods are based on robust optimization, aiming to maximize adversarial perturbations while minimizing empirical risk. The outer optimization problem in AT is often noisier and more ill-conditioned than standard empirical risk minimization, because each update is computed on adversarially perturbed inputs produced by an inner maximization procedure. In this regime, optimizer-induced geometry can have a non-negligible effect on both clean accuracy and robust accuracy. Adaptive optimizers such as Adam and AdamW rescale each coordinate of the update by an estimate of its gradient second moment. The Stochastic Gradient Descent (SGD) optimizer and its momentum variants avoid this per-coordinate rescaling, and have been observed to yield greater robustness under the high-variance gradients of AT dabouei2022revisiting . Empirical studies have also validated their superiority in AT pang2021bag . Some studies have also focused on enhancing robustness by exploring the Sharpness-Aware Minimization (SAM) method, which perturbs model weights to seek a flatter loss landscape zhang2024duality ; zhou2025sharpness . These studies make the parameter dynamics of min-max optimization analytically tractable, allowing robustness gains to be attributed to specific update-rule properties (e.g., implicit regularization of curvature or spectral norms) rather than treated as emergent artifacts. A previous study pang2021bag found that the effectiveness of using Adam kingma2015adam or AdamW loshchilov2017decoupled in AT is suboptimal. Recently, the Muon optimizer jordan2024muon has been introduced, which avoids expensive high-order projection algorithms while exploiting orthogonalized update geometry and enables a large-batch-friendly update that can be efficiently approximated by a small fixed number of Newton-Schulz iterations. This optimization method constrains the update direction (the polar factor of the momentum-accumulated gradient) to the Stiefel manifold, yielding orthogonalized, spectrally balanced parameter updates (though the weights W_{t} themselves need not lie on the manifold) liu2025muon . The Muon optimizer differs from other optimizers in that it orthogonalizes matrix-valued updates. For each 2D weight tensor, the Muon optimizer transforms the momentum update into its approximate polar factor, which can be realized via Newton–Schulz iterations. The updated geometry is fundamentally different from SGD robbins1951stochastic ; bottou2012stochastic and AdamW loshchilov2017decoupled . However, the impact of this optimizer on adversarial robustness remains unexplored. To fill this research gap, we carry out a systematic theoretical and experimental study of the Muon optimizer in the AT paradigm, as illustrated in Figure 1 . We first establish a theoretical mechanism and explanation linked to robustness. Intuitively, adversarial vulnerability is governed by norms of input gradients and Jacobians jakubovitz2018improving . We show that Muon’s polar update induces a spectral-norm stability ceiling at the update level. Each matrix-valued update has a controlled operator norm after orthogonalization. This property does not directly enforce small spectral norms of the learned weights. Instead, it provides a mechanism for limiting abrupt spectral growth in the optimization trajectory, offering a diagnostic explanation for the robustness behavior observed in our experiments. Empirically, we first adapt the Muon optimizer for AT, then conduct comprehensive experiments across multiple datasets and model architectures, to evaluate the robustness of four widely used optimizers: SGD, AdamW, SAM, and Muon. Across this grid, we observe that optimizer choice can induce large and consistent shifts in the robustness frontier: the same training recipe can yield markedly different robustness profiles depending on optimization dynamics. In particular, the robustness gains provided by the Muon optimizer are significantly greater than those of AdamW, and Muon’s performance is competitive with SGD across multiple settings. Compared with SAM, Muon should be viewed as an optimizer-geometry alternative rather than a direct replacement. It improves update geometry without introducing an additional weight-perturbation step, but it does not uniformly dominate SAM in robustness. To summarize, our key empirical findings are: (1) Muon constrains the update geometry: each matrix-valued update has a bounded operator norm after polar orthogonalization. This does not imply that Muon always produces smaller weight spectral norms than SGD, but it helps prevent the unstable spectral growth often observed with adaptive optimizers such as AdamW. (2) On small-scale datasets like CIFAR-10 krizhevsky2009learning , Muon is on par with or superior to SGD on overparameterized CNNs zagoruyko2016wide . On ViTs dosovitskiy2021image , Muon lags behind SGD yet remains markedly more stable and/or outperforms AdamW. However, on large-scale datasets like ImageNet deng2009imagenet , SGD still performs better, suggesting that sensitivity to large-scale learning rates is a key bottleneck, deserving future investigations. (3) Muon is an efficient optimizer-geometry alternative rather than a strict replacement for SAM. Figure 1 : The framework of this study. The left part of the figure is adapted from the work jordan2024muon . Our contributions are summarized as follows: • Revisiting optimizer choice as a security-relevant factor in AT. We identify orthogonalized matrix-update geometry as an underexplored factor in adversarial training and study Muon as a representative optimizer beyond the conventional SGD/AdamW/SAM comparison. • Mechanistic analysis of Muon’s spectral update geometry. We analyze Muon’s polar update rule and show that it imposes an operator-norm ceiling on matrix-valued updates. We further relate this update-level constraint to spectral growth and nuclear-norm descent under adversarial loss dynamics, clarifying both what Muon can and cannot guarantee. • Controlled multi-norm robustness evaluation. We conduct a controlled empirical comparison of SGD, AdamW, and

Read the complete text