Rethinking Language Model Scaling under Transferable Hypersphere Optimization

Liliang Ren, Yang Liu, Yelong Shen, Weizhu Chen · 2026

Read Rethinking Language Model Scaling under Transferable Hypersphere Optimization by Liliang Ren, Yang Liu, Yelong Shen, Weizhu Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28743.txt.

Introduction

Ren and co-authors introduce HyperP, a framework for transferring optimal learning rates under hypersphere optimization. Hypersphere methods fix weight matrix norms, which can stabilize training at scale. HyperP works with the Muon optimizer under a Frobenius-sphere constraint. It transfers learning rates across width, depth, token counts, and Mixture-of-Experts granularity. One base learning rate tuned at the smallest scale transferred to all larger budgets. HyperP gave 1.58 times compute efficiency over a strong Muon baseline.

Catalog record

CREATORLiliang Ren, Yang Liu, Yelong Shen, Weizhu Chen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66076
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.28743

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking Language Model Scaling under Transferable Hypersphere Optimization Liliang Ren, Yang Liu, Yelong Shen, Weizhu Chen Abstract Scaling laws for large language models depend critically on the optimizer and parameterization. Existing hyperparameter transfer laws are mainly developed for first-order optimizers, and they do not structurally prevent training instability at scale. Recent hypersphere optimization methods constrain weight matrices to a fixed-norm hypersphere, offering a promising alternative for more stable scaling. We introduce HyperP (Hypersphere Parameterization), the first framework for transferring optimal learning rates across model width, depth, training tokens, and Mixture-of-Experts (MoE) granularity under the Frobenius-sphere constraint with the Muon optimizer. We prove that weight decay is a first-order no-op on the Frobenius sphere, show that Depth- μ \mu P remains necessary, and find that the optimal learning rate follows the same data-scaling power law with the “magic exponent” 0.32 previously observed for AdamW. A single base learning rate tuned at the smallest scale transfers across all compute budgets under HyperP, yielding 1.58 × 1.58\times compute efficiency over a strong Muon baseline at 6 × 10 21 6\times 10^{21} FLOPs. Moreover, HyperP delivers transferable stability: all monitored instability indicators, including Z Z -values, output RMS, and activation outliers, remain bounded and non-increasing under training FLOPs scaling. We also propose SqrtGate, an MoE gating mechanism derived from the hypersphere constraint that preserves output RMS across MoE granularities for improved granularity scaling, and show that hypersphere optimization enables substantially larger auxiliary load-balancing weights, yielding both strong performance and good expert balance. We release our training codebase at https://github.com/microsoft/ArchScale . 1 Introduction Neural scaling laws [ 21 , 18 , 41 ] are central to the compute-efficient development of Large Language Models (LLMs) [ 34 , 6 , 28 , 16 , 10 , 40 , 48 ] . In practice, architectural designs and data recipes are explored at small scales for cost-savings, hoping that the improvements will persist when scaled up to prohibitively expensive compute budgets. However, identifying the true scaling behavior requires each model along the curve to be trained with near-optimal hyperparameters for its own scale. Even with well-tuned hyperparameters, scaling up training FLOPs routinely triggers logit explosion, activation outliers, and loss spikes [ 53 , 13 , 8 , 32 , 31 ] that push training off the optimal trajectory or even lead to training failures [ 52 , 35 ] . Existing hyperparameter transfer frameworks [ 49 , 50 , 5 , 24 , 7 ] primarily focus on transferring optimal Learning Rate (LR), weight decay, and batch size across width and depth, while largely overlooking how learning rate should scale with training tokens for transfer across training FLOPs. Moreover, they do not provide structural guarantees on training stability at larger scales. In practice, mitigations such as z-loss regularization [ 53 ] and careful weight decay scheduling [ 11 ] are applied as ad hoc patches rather than principled solutions. Recent work on hypersphere optimization [ 4 , 44 , 46 ] offers a fundamentally different approach. By constraining weight matrices to a fixed-norm sphere, hypersphere optimization provides structural stability guarantees: the weight-norm constraint naturally bounds output logit magnitudes for each linear projection. It also has the potential to eliminate weight decay, a notorious hyperparameter whose optimal value depends intricately on learning rate, training duration [ 43 , 3 ] , and model width [ 7 ] . In this work, we derive the first learning rate transfer laws across training FLOPs for hypersphere optimization, covering width, depth, training duration, and MoE granularity, under a typical second-order hypersphere optimizer MuonH [ 44 ] . We summarize our transfer laws as HyperP (Hypersphere Parameterization), a framework under which a single base learning rate tuned at the smallest scale transfers optimally to all compute budgets. Our theoretical and empirical results reveal that hypersphere optimization, when equipped with proper transfer laws, achieves not only optimal scaling efficiency but also transferable stability : the same hyperparameters that work at small scale produce equally or more stable training dynamics at large scale. With these results, fair comparisons of architectural scaling become possible: every model at every scale is trained at its transferred optimal learning rates, so the resulting scaling curves reflect each architecture’s near-optimal performance. Our contributions are as follows: • First transfer laws across FLOPs for hypersphere optimization. We derive HyperP, which achieves optimal LR transfer across training FLOPs, spanning width, depth, training tokens, and MoE granularity. We prove that weight decay is a first-order no-op under Frobenius-sphere optimization, and empirically show that removing weight decay does not harm model quality. We also derive that Depth- \mu P [ 50 ] is still required, disproving the claim that MuonH is inherently depth-transferable [ 44 ] , and discover the “magic exponent” 0.32 for data scaling, matching the previous result on AdamW [ 5 ] and suggesting universality across optimizers. • Transferable stability. Empirically, we show that HyperP yields stability transfer : all six monitored instability indicators ( Z -values, output RMS, activation outlier percentages for both attention and MoE sub-layers) are bounded and non-increasing as we scale training FLOPs for MoE models from 913M to 13.3B total parameters. • Robust MoE scaling and load balancing. We derive SqrtGate, a square-root gating mechanism that preserves output RMS across MoE granularities, reducing router Z -value peaks by 5\times compared to standard gating. HyperP’s optimal LR transfers robustly across MoE sparsity ( S\in\{1,\ldots,32\} ) and granularity ( k\in\{2,\ldots,64\} ). It also allows for much larger auxiliary load balancing weights, achieving the best validation loss and expert balance simultaneously, in contrast to prior findings that a large load balancing weight hurts model quality [ 42 ] . • Scalable compute efficiency leverage. A single base LR tuned at the smallest scale with 208M active parameters transfers effectively to all compute budgets explored. At the largest 6\times 10^{21} FLOPs, HyperP achieves 1.58\times Compute Efficiency Leverage (CEL) [ 21 , 41 ] over a strong Muon baseline with \mu P++ and weight decay scaling for dense models, and our MoE model with S=8,k=8 further achieves 3.38\times CEL over the dense models. The advantage of HyperP over the baseline grows monotonically with scale, implying even larger gains at frontier compute. • Architecture comparison under optimal scaling. With HyperP, we systematically examine dense (QK-Norm, Gated Attention) and MoE (SqrtGate, shared expert) architectures at their optimal performance, revealing that while the performance gains of SqrtGate and Gated Attention shrink as the training FLOPs increase, they provide significant stability gains that can remove the RMS spikes and control the exploding Z-values. 2 Background Hypersphere optimization. Hypersphere optimization constrains weight matrices to lie on a unit sphere under a chosen matrix norm. After each gradient update, the weight matrix W is re-projected as follows, W\leftarrow C\frac{W-\eta\,G}{\|W-\eta\,G\|}, (1) where \eta is learning rate, C is a constant, G denotes the weight update and \|\cdot\| is the chosen matrix norm. Several choices of the matrix norm have been proposed recently: MuonH [ 44 ] uses the Frobenius norm with the Muon optimizer [ 20 ] ; Both MuonSphere [ 46 ] and SSO [ 46 ] use the spectral norm, while SSO further applies the steepest descent on the spectral sphere. Previous works have explored column-

Read the complete text