MuCon: Clipped Muon Updates for LLM Training

Albert Yi · 2026

Read MuCon: Clipped Muon Updates for LLM Training by Albert Yi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26459.txt.

Introduction

Albert Yi studies MuCon, a clipped Muon variant for language models. Muon replaces a matrix update with its polar factor, mapping nonzero values to one. MuCon clips singular values at a threshold, leaving small ones alone, capping large ones, a spectral-norm-ball projection. Yi asks when that clipping can be approximated without a full singular-value decomposition. Identities point to a rational Newton filter, but near-threshold values make sign choices and rational solves ill-conditioned. Matrix-function tricks thus need stable primitives or regularization nearby.

Catalog record

CREATORAlbert Yi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34599
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26459

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MuCon: Clipped Muon Updates for LLM Training Albert Yi Abstract Muon-style optimizers take a matrix-valued momentum or preconditioned update B = U ​ diag ​ ( σ 1 , … , σ r ) ​ V ⊤ B=U{\rm diag}(\sigma_{1},\dots,\sigma_{r})V^{\top} and replace it with its canonical partial polar factor Polar ⁡ ( B ) = U ​ V ⊤ . \Pol(B)=UV^{\top}. This maps every nonzero singular value to one. MuCon is the clipped-Muon variant studied here: it applies singular-value clipping to the same Muon matrix, D τ MuCon ​ ( B ) = MClip τ ⁡ ( B ) = U ​ diag ​ ( min ⁡ { σ i , τ } ) ​ V ⊤ , τ > 0 . D^{\mathrm{MuCon}}_{\tau}(B)=\MClip_{\tau}(B)=U{\rm diag}(\min\{\sigma_{i},\tau\})V^{\top},\qquad\tau>0. Thus, MClip τ \MClip_{\tau} denotes the mathematical clipping operator, while MuCon denotes the optimizer primitive that substitutes this clipped direction for Muon’s polar direction. The Muon/MuCon scaling parameterization used in this work is called SpectralP : it is the hidden-matrix scaling recipe under which polar Muon or clipped MuCon directions are applied. The map MClip τ \MClip_{\tau} is the Frobenius projection onto the spectral-norm ball of radius τ \tau : it leaves singular values at or below τ \tau unchanged and modifies only the violating singular directions. This paper asks when the MuCon clipping step can be approximated without a full dense SVD. We record two exact identities, a polar/absolute-value formula and a scalar-root formulation leading to a rational Newton filter for the clipped positive-semidefinite factor, and identify the numerical obstruction common to both: singular values near the threshold make sign decisions and rational solves ill-conditioned. Matrix-function methods are therefore useful only when paired with stable polar/square-root primitives or explicit regularization near the clipping boundary. 1 Introduction Many optimizer designs modify matrix-valued updates through spectral transformations. A well-known example is Muon-style orthogonalization. If B_{t}=U\Sigma V^{\top} is a compact SVD of the matrix-valued momentum or preconditioned update passed to Muon’s matrix step, the mathematical target in this report is the canonical partial polar factor D_{t}^{\mathrm{Muon}}=\Pol(B_{t})=UV^{\top}. For rank-deficient matrices, \Pol denotes this SVD-defined partial isometry, not an arbitrary orthogonal completion. In implementations it is often approximated by a Newton-Schulz iteration. The operation is aggressive: every nonzero singular value is replaced by one. This report studies the more selective map \MClip_{\tau}(M)=U{\rm diag}(\min\{\sigma_{i},\tau\})V^{\top},\qquad M=U{\rm diag}(\sigma_{i})V^{\top}, so that \sigma_{i}(\MClip_{\tau}(M))=\min\{\sigma_{i}(M),\tau\}. The default threshold is \tau=1 , and we write \MClip(M)=\MClip_{1}(M) when no threshold is shown. Clipping preserves all singular values at or below \tau and modifies only the violating directions. When this map replaces Muon’s polar step on the Muon matrix B_{t} , the resulting clipped-Muon update is D_{t}^{\mathrm{MuCon}}=\MClip_{\tau_{t}}(B_{t}),\qquad D_{t}^{\mathrm{Muon}}=\Pol(B_{t}). Throughout the paper, \MClip_{\tau} denotes the mathematical clipping operator, while MuCon denotes the optimizer primitive that applies this operator inside the Muon update pipeline. Separately, SpectralP denotes the scaling parameterization used for the Muon/MuCon hidden-matrix groups in this work; it is not a new clipping map. The operator \MClip_{\tau} is exactly the Frobenius projection onto the spectral-norm ball \mathcal{B}_{2}(\tau)=\{X\in\mathbb{R}^{m\times n}:\|X\|_{2}\leq\tau\}:\qquad\MClip_{\tau}(M)=\mathop{\mathrm{argmin}}_{X\in\mathcal{B}_{2}(\tau)}\frac{1}{2}\|X-M\|_{F}^{2}. The exact algorithm is simple: compute an SVD, clip the singular values, and reconstruct the matrix. Its dense cost, O(mn\min(m,n)), is usually too high for repeated use inside an optimizer; see Golub and Van Loan 2013 for standard background on dense matrix decompositions. The central numerical question is therefore: Can the MuCon clipping step be approximated accurately enough for optimizer use while avoiding a full dense SVD? A key structural identity already suggests the correct regime split. Let \mathcal{I}_{>}=\{i:\sigma_{i}(M)>\tau\},\qquad k_{>}=|\mathcal{I}_{>}|. Then \MClip_{\tau}(M)=M-U_{>}{\rm diag}\bigl((\sigma_{i}-\tau)_{i\in\mathcal{I}_{>}}\bigr)V_{>}^{\top}, where U_{>},V_{>} contain only the singular vectors whose singular values exceed \tau . Hence, clipping is a rank- k_{>} correction to M . When k_{>} is small, a partial SVD, Lanczos method, or randomized subspace method is the most selective baseline; when k_{>} is large, global matrix-function iterations may be competitive. Contributions. The report makes three technical points for SpectralP MuCon. First, it separates the mathematical clipping map from the clipped-Muon optimizer primitive and records the projection and low-rank correction identities that any approximation should respect. Second, it derives a polar/absolute-value formulation of clipping and explains why threshold eigenvalues are numerically delicate. Third, it analyzes a rational Newton iteration for the clipped positive-semidefinite factor and clarifies that it is a spectral filter, not a standalone SVD-free algorithm. 2 Background: SpectralP and Width-Depth Scaled Training This project is motivated by hyperparameter transfer under simultaneous width and depth scaling. CompleteP studies joint width-depth transfer and non-lazy feature learning in deep Transformers ( Dey et al. 2025 ) . Spectral \mu\mathrm{P} and related operator-norm viewpoints motivate controlling weights and updates in normalized spectral norm ( Yang et al. 2023 ; Zheng et al. 2026 ) . In this report, the resulting hidden-matrix scaling parameterization for Muon and MuCon is called SpectralP . SpectralP assigns hidden two-dimensional matrix groups to a spectral update class, while scalar, vector, embedding, and unembedding groups remain AdamW companion groups. Both viewpoints point to the same numerical need: matrix updates should have controlled spectra without requiring expensive decompositions at every optimizer step. 2.1 Hyperparameter Transfer and Maximal-Update Parameterization A central practical problem in large-model training is hyperparameter transfer. Ideally, one tunes hyperparameters such as learning rate, initialization scale, and weight decay on a small model and transfers them to a much larger one. If the same base hyperparameters remain close to optimal after scaling, one obtains a “tune small, train large” strategy. Maximal update parameterization, or \mu\mathrm{P} , was introduced to make this possible under width scaling ( Yang and Hu 2021 ; Yang et al. 2022 ) . In its simplest form, \mu\mathrm{P} aims to preserve nontrivial feature learning as the width N grows. If h_{\ell}(x)\in\mathbb{R}^{d_{\ell}} is the hidden representation at layer \ell , the invariant scale is coordinate-wise, or equivalently RMS-normalized: \|h_{\ell}(x)\|_{\mathrm{R},d_{\ell}}=\Theta(1),\qquad\|\Delta h_{\ell}(x)\|_{\mathrm{R},d_{\ell}}=\Theta(1),\qquad\|a\|_{\mathrm{R},d}:=\frac{\|a\|_{2}}{\sqrt{d}}. Thus, the parameterization should avoid both lazy dynamics, \|\Delta h_{\ell}(x)\|_{\mathrm{R},d_{\ell}}\to 0, and unstable dynamics, \|\Delta h_{\ell}(x)\|_{\mathrm{R},d_{\ell}}\to\infty. For width-only scaling, a typical hidden-matrix AdamW learning-rate rule takes the form \eta_{\mathrm{hidden}}=\eta_{\mathrm{base}}m_{N}^{-1},\qquad m_{N}=\frac{N}{N_{\mathrm{base}}}. The exact exponent depends on the optimizer and parameterization, but the principle is that matrix updates must be rescaled so that their induced feature movement remains order one. Modern foundation models do not scale only in width; they also become deeper. A useful parameterization should therefore preserve transfer as N\to\infty,\qquad L\to\infty, where L is the number of residual blocks. 2.2 CompleteP: Residual Scaling for Deep Transformers CompleteP studies joint width-depth scaling for pre-LN decoder-only Transformer language models ( Dey et al. 2025 ) . Its starting point is the residual recursion h_{\ell+1}=h_{\ell}+L^{-\alpha}F_{\ell}(h_{\ell}),\qquad\ell=1,\dots,L, where F_{\ell} is a residual block, such as an attention or MLP block. The exponent \alpha\in[1/2,1] controls the depth scaling of each residual branch. The two most important cases are \alpha=\frac{1}{2}\qquad\text{and}\qquad\alpha=1. The choice \alpha=1/2 is natural from initialization stability. If residual increments are roughly independent and have comparable size, then the accumulated residual variance scales as \sum_{\ell=1}^{L}L^{-2\alpha}=L^{1-2\alpha}. Avoiding variance explosion, therefore, requires \alpha\geq\frac{1}{2}. CompleteP argues that initialization stability is not enough. It advocates the stronger scaling \alpha=1, so that h_{\ell+1}=h_{\ell}+L^{-1}F_{\ell}(h_{\ell}). In practical scaling experiments one writes m_{N}=\frac{N}{N_{\mathrm{base}}},\qquad m_{L}=\frac{L}{L_{\mathrm{base}}}, and uses h_{\ell+1}=h_{\ell}+m_{L}^{-1}F_{\ell}(h_{\ell}). 2.2.1 AdamW scaling in CompleteP CompleteP is not only a residual multiplier. It also specifies how model and optimizer hyperparameters should scale with m_{N} and m_{L} . For hidden matrix weights,

Read the complete text