AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Training
Zhiyuan Li, Yuan Wu, Yi Chang · 2026
Read AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Training by Zhiyuan Li, Yuan Wu, Yi Chang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11864.txt.
Introduction
Zhiyuan Li, Yuan Wu, and Yi Chang aim to steady the training of large language models. A common fix for wild gradients is to clip them to a set size. The usual fix treats all parts of the model as one block.
That can hurt, since wild parts force cuts on calm parts too. The team call this spill-over. Their method, named AGGC in the work, clips each group of parts by its own scale. Group-wise clips could keep training calm with less harm to calm parts. The work tests this idea on large model runs.
Catalog record
| CREATOR | Zhiyuan Li, Yuan Wu, Yi Chang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-11864
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AGGC: Adaptive Group Gradient Clipping for Stabilizing Large Language Model Training Zhiyuan Li, Yuan Wu, Yi Chang Abstract To stabilize the training of Large Language Models (LLMs), gradient clipping is a nearly ubiquitous heuristic used to alleviate exploding gradients. However, traditional global norm clipping erroneously presupposes gradient homogeneity across different functional modules, leading to an adverse "spill-over" effect where volatile parameters force unnecessary scaling on stable ones. To overcome this, we propose Adaptive Group-wise Gradient Clipping (AGGC). AGGC partitions parameters into groups based on functional types and regulates each according to its historical behavior using an Exponential Moving Average (EMA). Specifically, it constructs an adaptive interval to simultaneously mitigate gradient explosion and vanishing, while employing a time-dependent scheduling mechanism to balance exploration and convergence. Experiments on LLaMA 2-7B, Mistral-7B, and Gemma-7B models show that AGGC consistently outperforms LoRA and frequently surpasses Full Fine-Tuning. On the GSM8K benchmark, Mistral-7B fine-tuned with AGGC achieves an accuracy of 72.93%, exceeding LoRA’s 69.5%. AGGC also effectively stabilizes Reinforcement Learning with Verifiable Rewards (RLVR), enhancing the logic deduction of Qwen 2.5 and Llama 3.2 models. Experimental results demonstrate that AGGC effectively addresses the limitations of traditional gradient clipping methods, particularly in overcoming gradient heterogeneity, by utilizing a modular, adaptive clipping strategy to stabilize the training process. Due to its lightweight design, AGGC can be seamlessly integrated into existing post-training pipelines with negligible overhead 1 1 1 Code is available at: https://github.com/ZhiyuanLi218/AGGC . 1 Introduction The relentless scaling of Large Language Models (LLMs) in terms of parameter count and depth has fundamentally reshaped the deep learning landscape, enabling unprecedented capabilities across diverse tasks ( 17 ) . However, this scaling progression has simultaneously exacerbated foundational challenges in optimization, particularly concerning training stability. During both pre-training and post-training pipelines, such as Supervised Fine-Tuning (SFT) and Reinforcement Learning with Human Feedback (RLHF), LLMs frequently encounter catastrophic loss spikes ( 26 ; 21 ; 16 ) . These spikes indicate transient or persistent numerical instability, which undermines training efficiency and compromises the quality of the resultant model convergence. Gradient-based optimization necessitates a sophisticated trade-off between rapid convergence and numerical stability. As a canonical technique for safeguarding training stability, gradient clipping was originally developed to alleviate the critical issue of exploding gradients ( 18 ) . By capping the magnitude of gradient updates, this method constrains the parameter updates of gradient descent within a rational range. Owing to its validated effectiveness as a regularization strategy, gradient clipping, particularly in the form of global norm clipping, has evolved into a nearly ubiquitous and indispensable heuristic for the training of contemporary deep neural networks ( 13 ; 14 ; 2 ; 11 ) . This assumption leads to the "spill-over" effect, where gradients from volatile modules with large magnitudes unnecessarily scale stable modules, disrupting the stability of the training process. Despite its ubiquity, conventional global gradient clipping bears inherent limitations rooted in transformer-based LLMs’ architectural traits. This standard method computes a unified \ell_{2} norm across all parameters and erroneously presupposes gradient homogeneity—i.e., comparable magnitudes and variances across all functional modules ( 27 ; 34 ; 30 ) . Figure 1: Grad norm of Up module over training steps. LLMs exhibit marked architectural heterogeneity: parameter groups (Query/Key/Value projections, Layer Normalization, Feed-Forward Networks) possess divergent gradient dynamics ( 26 ; 23 ) . Empirically, gradient magnitudes vary drastically across components, with certain modules generating dominant large gradients ( 27 ) , rendering universal clipping inherently suboptimal—especially for models with heterogeneous loss curvatures ( 34 ) . To address the inherent limitations of static global gradient clipping strategies, we propose a novel method termed Adaptive Group-wise Gradient Clipping (AGGC). The core design principle of AGGC lies in disentangling the optimization constraints across distinct functional components of the model. Specifically, the gradient regulation mechanism for each component group is formulated as a dynamic process. AGGC leverages an Exponential Moving Average (EMA) to track the historical gradient norms of individual groups, generating a smoothed scaling statistic denoted as S_{j}^{(t)} , where j represents the group and t denotes the training step. Furthermore, AGGC introduces a bidirectional admissible interval defined by [\mathrm{L}_{j}^{(t)},\mathrm{U}_{j}^{(t)}] for each group j at training step t . The upper bound \mathrm{U}_{j}^{(t)} is designed to suppress gradient explosion, while the lower bound \mathrm{L}_{j}^{(t)} proactively alleviates the risks of gradient vanishing and premature stagnation of parameter updates. Critically, the width of this interval is modulated by a time-dependent scheduling strategy. Specifically, the bounds \mathrm{L}_{j}^{(t)} and \mathrm{U}_{j}^{(t)} are controlled by the multiplicative coefficients \alpha_{\mathrm{low}}^{(t)} and \alpha_{\mathrm{high}}^{(t)} , which define the lower and upper limits of the gradient norm at each training step t . In the early training phase, these coefficients are set relatively high to allow for sufficient parameter exploration, and they gradually decrease to stabilize the model’s convergence in later stages. As illustrated in Figure 1 , AGGC maintains a smoother and more stable optimization trajectory compared to the fluctuations observed in LoRA. The AGGC framework contributes significantly to the field of optimization by providing a robust, nuanced control mechanism tailored for LLMs: 1. Functional Grouping and Localized Regulation: We formalize a Group-wise gradient regulation strategy, partitioning parameters based on functional module type to align optimization control with architectural heterogeneity. This design successfully eliminates the adverse spill-over effect inherent in global clipping, ensuring efficient and unbiased utilization of small-scale gradient signals. 2. Adaptive and Bi-directional Control: AGGC leverages EMA to dynamically estimate group-specific gradient scales, establishing an adaptive, two-sided admissible interval [\mathrm{L}_{j}^{(t)},\mathrm{U}_{j}^{(t)}] . This approach actively mitigates both exploding gradients and update collapse, a capability largely absent in standard clipping methods. 3. Integration of Magnitude Scheduling: We introduce a time-dependent, linear scheduling mechanism for the multiplicative coefficients ( \alpha_{\mathrm{low}}^{(t)},\alpha_{\mathrm{high}}^{(t)} ) that define the bounds. This provides a principled, dedicated method for gradient magnitude control scheduling, optimally balancing exploratory dynamics early in training with stable convergence in later stages. 2 Related Work The theoretical foundation of traditional gradient clipping originated from the in-depth analysis of training difficulties in recurrent neural networks (RNNs). ( 18 ) presented a seminal work that systematically explored gradient vanishing and explosion issues from analytical, geometric, and dynamic system perspectives. They proposed gradient norm clipping to mitigate exploding gradients and soft constraint methods for vanishing gradients, laying the theoretical groundwork for all subsequent gradient clipping approaches. Global norm clipping’s core idea is to limit the maximum magnitude of gradients using a unified global threshold. This effectively prevents numerical instability and gradient explosion ( 13 ) . Nevertheless, traditional global clipping exhibits inherent limitations. ( 13 ) pointed out in their comprehensive review that despite its simplicity and widespread adoption, gradient clipping mechanisms typically require specific threshold values c and strong noise assumptions to guarantee convergence. Another pivotal observation came from ( 33 ) , who found that gradient smoothness exhibits significant variability along the training trajectory of deep neural networks, and this smoothness is positively correlated with gradient norm. To overcome the limitations of traditional global clipping, researchers have developed a series of adaptive clipping techniques that dynamically adjust strategies based on gradient characteristics during training. Early adaptive methods such as AdaGrad ( 6 ) and RMSProp indirectly addressed gradient adaptation by tracking historical gradient information to adjust learning rates dynamically. However,