Can Muon Fine-tune Adam-Pretrained Models?

Xingyu Qu, Peigeng Huang, Samuel Horvath · 2026

Read Can Muon Fine-tune Adam-Pretrained Models? by Xingyu Qu, Peigeng Huang, Samuel Horvath (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10468.txt.

Introduction

Xingyu Qu and two colleagues ask whether Muon can fine-tune models pretrained with Adam. Muon offers an efficient pretraining alternative to Adam, yet sees little use in fine-tuning. Most open models come pretrained with Adam, and moving to Muon for fine-tuning without care degrades performance. Their 2026 paper investigates that mismatch through controlled experiments.

The authors relate the mismatch to the distinct implicit biases of Adam and Muon. Evidence shows the mismatch disrupts pretrained knowledge in proportion to update strength. That suggests constraining updates should mitigate the mismatch. The authors test it with LoRA, which freezes pretrained weights and holds updates inside a low-rank subspace. Across language and vision tasks, LoRA shrinks the performance gap between Adam and Muon seen under full fine-tuning. Studies on LoRA rank, forgetting, and LoRA variants confirm mismatch severity tracks update strength.

Catalog record

CREATORXingyu Qu, Peigeng Huang, Samuel Horvath
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10468

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can Muon Fine-tune Adam-Pretrained Models? Xingyu Qu, Peigeng Huang, Samuel Horvath Abstract Muon has emerged as an efficient alternative to Adam for pretraining, yet remains underused for fine-tuning. A key obstacle is that most open models are pretrained with Adam, and naively switching to Muon for fine-tuning leads to degraded performance due to an optimizer mismatch. We investigate this mismatch through controlled experiments and relate it to the distinct implicit biases of Adam and Muon. We provide evidence that the mismatch disrupts pretrained knowledge, and that this disruption scales with update strength. This leads us to hypothesize that constraining updates should mitigate the mismatch. We validate this with LoRA: across language and vision tasks, LoRA reduces the performance gap between Adam and Muon observed under full fine-tuning. Studies on LoRA rank, catastrophic forgetting, and LoRA variants further confirm that mismatch severity correlates with update strength. These results shed light on how optimizer mismatch affects fine-tuning and how it can be mitigated. Our code is available here . † † affiliationnotice: Equal contribution 1 Introduction Muon ( 29 ) ( M oment U m O rthogonalized by N ewton-Schulz) has emerged as a promising alternative to Adam ( 33 ; 44 ) for large language model (LLM) pretraining. It orthogonalizes the momentum matrix before each update, achieving approximately 2\times compute efficiency over Adam ( 42 ; 58 ) while requiring less memory by eliminating the second moment. Notably, it has been successfully adopted for training state-of-the-art models up to the trillion-parameter scale, including Kimi K2/2.5 ( 66 ) and GLM-4.5/4.7 ( 78 ) . Figure 1 : Relative perplexity (normalized by pretrained baseline) during full fine-tuning on NanoChat ( 32 ) . Fine-tuning with a mismatched optimizer (e.g., using Muon on an Adam-pretrained model) consistently results in worse perplexity. See Section 3 for details. Despite these successes, existing work on Muon has focused almost exclusively on pretraining, leaving fine-tuning, the dominant training paradigm, largely unexplored. Preliminary results from 42 reveal an optimizer mismatch problem: applying Muon to fine-tune an Adam-pretrained model yields suboptimal results compared to Adam, and vice versa. We illustrate this in Figure 1 . Since most open models are pretrained with Adam, this mismatch severely limits Muon’s practical applicability. Understanding and addressing this mismatch is therefore critical. This work presents the first in-depth analysis of the optimizer mismatch problem, combining empirical exploration with theoretical insights. We first reproduce the phenomenon through controlled experiments and relate it to the distinct implicit biases of Adam and Muon, which produce pretrained weights with different structural properties. We find that mismatch increases sensitivity to update strength during fine-tuning, suggesting that it degrades performance by disrupting pretrained knowledge. Based on this analysis, we hypothesize that constraining the extent of updates should mitigate the mismatch. We examine this hypothesis with Low-Rank Adaptation (LoRA) ( 26 ) , which freezes the pretrained weights and restricts updates to a low-rank subspace, thereby limiting how much the fine-tuning optimizer can alter them. We verify this through extensive experiments on language and vision benchmarks, showing that LoRA combined with Muon (LoRA-Muon) matches or outperforms its Adam counterpart (LoRA-Adam). We further validate our hypothesis through rank studies, catastrophic forgetting ( 46 ; 18 ) measurements, and investigation of LoRA variants. To summarize, we make the following contributions: • We reproduce and analyze the optimizer mismatch problem at accessible scales, relate it to the distinct implicit biases of Adam and Muon, and provide evidence that mismatch degrades performance by disrupting pretrained knowledge. • We show that constraining updates via LoRA mitigates this mismatch, enabling LoRA-Muon to perform on-par with LoRA-Adam across language and vision tasks. Studies on LoRA rank, catastrophic forgetting, and compatibility with LoRA variants further support this finding. 2 Preliminaries 2.1 Muon Optimizer Muon ( 29 ) is an optimizer designed for matrix-shaped parameters in neural networks, and is typically paired with Adam for non-matrix parameters such as embeddings and biases. In practice, Muon’s implementation varies slightly across frameworks, as detailed in Appendix B . We adopt the implementation of 42 (except in Section 3 ), who first reported the optimizer mismatch problem. Their implementation serves as the basis for MuonClip, the optimizer used to pretrain the 32B/1T Kimi K2 model ( 66 ) . Given a parameter matrix W_{t}\in\mathbb{R}^{m\times n} and its gradient G_{t} , Muon updates the parameters as: \displaystyle\textstyle O_{t}=\mathrm{NS}(M_{t}),\quad M_{t}=\beta M_{t-1}+G_{t}, (1) \displaystyle\textstyle W_{t+1}=W_{t}-\eta O_{t} (2) where \beta is the momentum coefficient, \eta is the learning rate, and \mathrm{NS}(\cdot) denotes the Newton-Schulz iteration that approximates the nearest semi-orthogonal matrix. Specifically, for the singular value decomposition M_{t}=U\Sigma V^{\top} , we have O_{t}\approx UV^{\top}=(M_{t}M_{t}^{\top})^{-1/2}M_{t} . This orthogonalization ensures that updates have nearly uniform singular values, effectively applying equal step sizes across all directions in the weight space. Later works, such as Polar Express (PE) ( 2 ) , replace the fixed Newton-Schulz coefficients with adaptive ones for a more accurate approximation. In contrast, Adam ( 33 ) is the dominant optimizer for both pretraining and fine-tuning large language models. It uses element-wise adaptive learning rates based on first- and second-moment estimates of the gradient: \displaystyle\textstyle M_{t}=\beta_{1}M_{t-1}+(1-\beta_{1})G_{t}, (3) \displaystyle\textstyle V_{t}=\beta_{2}V_{t-1}+(1-\beta_{2})G_{t}^{2}, (4) \displaystyle\textstyle W_{t+1}=W_{t}-\eta\cdot\hat{M}_{t}\oslash(\sqrt{\hat{V}_{t}}+\epsilon) (5) where \hat{M}_{t}=M_{t}/(1-\beta_{1}^{t}) and \hat{V}_{t}=V_{t}/(1-\beta_{2}^{t}) are the bias-corrected estimates, and \oslash denotes element-wise division. The key distinction between these two optimizers lies in their preconditioning: Adam adapts step sizes independently for each parameter via element-wise rescaling 1/(\sqrt{\hat{V}_{t}}+\epsilon) , whereas Muon adapts step sizes across singular directions of the gradient matrix via matrix-level preconditioning (MM^{\top})^{-1/2} . As we show in Section 3 , this difference leads to fundamentally different implicit biases, which in turn give rise to the optimizer mismatch problem. 2.2 Low-Rank Adaptation (LoRA) Low-Rank Adaptation (LoRA) ( 26 ) is a parameter-efficient fine-tuning method that freezes the pretrained weights and introduces trainable low-rank decomposition matrices. For a pretrained weight matrix W_{0}\in\mathbb{R}^{m\times n} , LoRA parameterizes the weight update as: W=W_{0}+\Delta W=W_{0}+\tilde{\alpha}BA (6) where B\in\mathbb{R}^{m\times r} and A\in\mathbb{R}^{r\times n} are the trainable low-rank matrices, r\ll\min(m,n) is the rank, and \tilde{\alpha} is a scaling factor typically set to \alpha/r ( 26 ) or \alpha/\sqrt{r} ( 30 ) , where \alpha is a hyperparameter. By default, B is initialized to zero and A is drawn from a random Gaussian, so that \Delta W=0 at the start of training. During fine-tuning, only A and B are updated while W_{0} remains frozen. This approach significantly reduces the number of trainable parameters and memory requirements. However, LoRA often underperforms full fine-tuning due to the low-rank constraint. Various variants have been proposed to narrow this gap, including initialization techniques that bring LoRA updates closer to full fine-tuning ( 81 ; 47 ; 65 ) . On the other hand, LoRA has been shown to “learn less but forget less,” suggesting that the low-rank constraint helps preserve pretrained knowledge ( 7 ) . 3 Analyzing Optimizer Mismatch 42 reported that fine-tuning with a mismatched optimizer—using Adam on Muon-pretrained models or vice versa—leads to degraded performance compared to using the same optimizer for both stages. This optimizer mismatch problem significantly limits the practical applicability of Muon for fine-tuning, since most publicly available pretrained models were trained with Adam. To understand this phenomenon, we conduct controlled experiments in a simplified setting. Experimental setup.

Read the complete text