GQA- μ \mu P: The Maximal Parameterization Update for Grouped Query Attention
Kyle R. Chickering, Huijuan Wang, Mengxi Wu, Alexander Moreno, Muhao Chen, Xuezhe Ma, Daria Soboleva, Joel Hestness, Zhengzhong Liu, Eric Xing · 2026
Read GQA- μ \mu P: The Maximal Parameterization Update for Grouped Query Attention by Kyle R. Chickering, Huijuan Wang, Mengxi Wu, Alexander Moreno, Muhao Chen, Xuezhe Ma, Daria Soboleva, Joel Hestness, Zhengzhong Liu, Eric Xing (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15290.txt.
Introduction
Kyle R. Chickering and nine others derive the maximal update parameterization for grouped-query attention, allowing tuning transfer across architectures. Maximal settings ensure transfer through sound analysis but are hard to derive for new designs. Raising spectral-norm rules from rule-of-thumb to definition yields complete depth and weight-decay scalings, and a tweaked spectral norm handles non-full-rank weights. Tests confirm learning-rate transfer across the grouped-query repetition parameter and over weight decay.
Catalog record
| CREATOR | Kyle R. Chickering, Huijuan Wang, Mengxi Wu, Alexander Moreno, Muhao Chen, Xuezhe Ma, Daria Soboleva, Joel Hestness, Zhengzhong Liu, Eric Xing |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53536 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15290
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GQA- μ \mu P: The Maximal Parameterization Update for Grouped Query Attention Kyle R. Chickering, Huijuan Wang, Mengxi Wu, Alexander Moreno, Muhao Chen, Xuezhe Ma, Daria Soboleva, Joel Hestness, Zhengzhong Liu, Eric Xing Abstract Hyperparameter transfer across model architectures dramatically reduces the amount of compute necessary for tuning large language models (LLMs). The maximal update parameterization ( μ \mu P) ensures transfer through principled mathematical analysis but can be challenging to derive for new model architectures. Building on the spectral feature-learning view of 26 , we make two advances. First, we promote spectral norm conditions on the weights from a heuristic to the definition of feature learning, and as a consequence arrive at the Complete-P depth and weight-decay scalings without recourse to lazy-learning. Second, we consider a modified spectral norm that preserves the valid scaling law of network weights when weight matrices are not full rank. This enables (to our knowledge, the first) derivation of μ \mu P scalings for grouped-query attention (GQA). We demonstrate the efficacy of our theoretical derivations by showing learning rate transfer across the GQA repetition hyperparameter as well as experiments regarding transfer over weight decay. 1 Introduction The maximal update parametrization ( \mu P) ( 25 ; 24 ) provides principled rules for zero-shot learning rate transfer across model widths. Thus, large terminal model hyperparameters can be determined by sweeping a small proxy model. \mu P has been used to train models up to at least 13B parameters with zero-shot transfer ( 6 ; 7 ; 20 ) . Its applicability, however, has been largely limited to learning rate transfer across model widths. To broaden this scope, 8 introduced Complete-P, extending the original prescriptions to weight decay and model depth. However, many common architectures that are widely deployed in production still lack established \mu P scalings. This paper seeks to close this gap by extending the spectral \mu P framework of 26 to be more practically useful in deriving \mu P prescriptions for novel architectures. As an example of the utility of our framework, we derive (to our knowledge, the first) \mu P scaling for grouped-query attention (GQA) ( 1 ) . Our analysis reveals that GQA surfaces several difficulties that prior work has left unaddressed. First, when using GQA the original \mu P implementation passes coordinate checks, i.e., the customary correctness tests for the implementation. However, empirical analysis shows that the original \mu P implementation fails to transfer learning rates, seemingly contradicting established theory (see Figures 1 and 5 ). We resolve this by extending the spectral-norm version of \mu P introduced in 26 , and showing that the original \mu P implementation does not pass a more rigorous spectral-norm coordinate check. Second, the intrinsic low rank of GQA weight matrices skews the expected size of layer outputs. To address this issue, we introduce a new norm, namely the expected operator norm, to replace the spectral norm in spectral \mu P theory and restore the desired scaling behavior. Our primary contributions are threefold: 1. We extend the spectral \mu P theory of 26 , which allows derivations of \mu P for more advanced architectures like weight decay, recursion blocks, and GQA. Our work provides, to our knowledge, the first derivation of \mu P scaling for GQA. 2. We perform empirical analysis to validate the theory and offer practical guidance for learning rate transfer across GQA settings. In particular, we suggest that transferring across different numbers of GQA repetitions leads to noisy transfer dynamics, suggesting caution when attempting to transfer learning rate. 3. Our experiments show that, with the correct scalings, both weight decay and the training-time constant \tau_{\text{epoch}} introduced in 22 appear to be transferable. Figure 1: Comparison of the standard parameterization (left), the vanilla Adam- \mu P parameterization (middle), and our GQA- \mu P scaling (right). For a fixed model size, we vary the number of KV heads. The dashed lines indicate the mean optimal learning rates for each parameterization, and the shaded grey region denotes the standard deviation of the optimal learning rates. All models are trained to 10 tokens per parameter (TPP). Additional details can be found in Appendix B.1.2 . 2 Related Work Foundations of \mu P: \mu P builds on a series of works by Yang, developing the Tensor Programs framework ( 28 ; 29 ; 30 ; 25 ; 24 ; 27 ) . This line of work uses random matrix theory to carefully analyze the mathematical properties of neural networks during training, while also empirically demonstrating that these theoretical approaches remain valuable for real-world deep learning. Within the framework of Tensor Programs, 24 derives the well-known \mu P scaling laws for width under SGD and Adam training. The final paper in the series 27 attempts to extend \mu P to depth scalings. However, they were unable to extend to the case of residual blocks with standard configurations for the hidden layers. Finally, the foundation of the mathematical framework presented in this work builds on 26 , who show an alternative derivation of the results in 24 based on spectral norms. Models using GQA: Group-query attention (GQA) ( 1 ) is an efficient attention mechanism that reduces memory usage by sharing key and value heads across groups of query heads. Due to its favorable trade-off between memory efficiency and model performance, GQA has been widely used in modern large language models, including Mistral 7B ( 14 ) , LLaMA 3 ( 11 ) , Qwen3 ( 23 ) and K2-V2 ( 17 ) . Extensions of \mu P: The original \mu P formulation presented in 24 applies only to scaling the width of a fixed depth, fixed batch size neural network. While already a powerful tool, later authors have sought to extend the principles of \mu P to cover cases not covered by the original formulation. 7 do large-scale validation experiments using \mu P and find empirical evidence that learning rate can transfer across batch and dataset size. 7 suggests \mu P-type scalings for weight decay, the Adam \varepsilon , and depth. Their contributions to depth scaling are most notable, as their empirical findings contradict the scaling presented in 27 . However, their extensive empirical analysis suggests that the scaling they derive is correct. We arrive at the same scaling in Section 3.3 using the framework we outline in this paper. More recently 19 extended the work of Dey et al. 8 by extending the SDE parameterization to cover hyperparameter transfer for batch size, as well as showing the value of per-layer learning rate tuning. 6 apply \mu P in the context of large-scale, low-precision LLM training. They use ABC parameterizations to apply the \mu P scaling rules while maintaining unit variance for all layers in the network, which they refer to as unit scaling- \mu P. Additionally, they empirically validate that learning rate transfer persists across datasets, batch sizes, depths, and training iterations under controlled conditions. 20 suggest a different, more simplified version of the unit scaling- \mu P which they also show works for training low-precision networks with \mu P. Finally, a related subsequent work 32 has proposed a similar theoretical framework to ours. Both our work and theirs share the perspective that the spectral norm