Imbalanced Gradients in RL Post-Training of Multi-Task LLMs
Runzhe Wu, Ankur Samanta, Ayush Jain, Scott Fujimoto, Jeongyeol Kwon, Ben Kretzu, Youliang Yu, Kaveh Hassani, Boris Vidolov, Yonathan Efroni · 2025
Read Imbalanced Gradients in RL Post-Training of Multi-Task LLMs by Runzhe Wu, Ankur Samanta, Ayush Jain, Scott Fujimoto, Jeongyeol Kwon, Ben Kretzu, Youliang Yu, Kaveh Hassani, Boris Vidolov, Yonathan Efroni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19178.txt.
Introduction
Runzhe Wu and nine co-authors show that gradient imbalance distorts reinforcement learning post-training when several tasks train a language model together. Mixing task datasets and optimizing them jointly assumes each task pushes the model with similar force, but the authors find some tasks produce far larger gradients than others.
Updates then lean toward the loud tasks, even though those tasks do not learn faster; some large-gradient tasks gain as little as, or less than, quiet ones. That mismatch means joint training can waste effort and starve tasks that need it. The finding pushes trainers to balance gradients deliberately rather than trust the mix.
Catalog record
| CREATOR | Runzhe Wu, Ankur Samanta, Ayush Jain, Scott Fujimoto, Jeongyeol Kwon, Ben Kretzu, Youliang Yu, Kaveh Hassani, Boris Vidolov, Yonathan Efroni |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19178
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Imbalanced Gradients in RL Post-Training of Multi-Task LLMs Runzhe Wu, Ankur Samanta, Ayush Jain, Scott Fujimoto, Jeongyeol Kwon, Ben Kretzu, Youliang Yu, Kaveh Hassani, Boris Vidolov, Yonathan Efroni Abstract Multi-task post-training of large language models (LLMs) is typically performed by mixing datasets from different tasks and optimizing them jointly. This approach implicitly assumes that all tasks contribute gradients of similar magnitudes; when this assumption fails, optimization becomes biased toward large-gradient tasks. In this paper, however, we show that this assumption fails in RL post-training: certain tasks produce significantly larger gradients, thus biasing updates toward those tasks. Such gradient imbalance would be justified only if larger gradients implied larger learning gains on the tasks (i.e., larger performance improvements)—but we find this is not true. Large-gradient tasks can achieve similar or even much lower learning gains than small-gradient ones. Further analyses reveal that these gradient imbalances cannot be explained by typical training statistics such as training rewards or advantages, suggesting that they arise from the inherent differences between tasks. This cautions against naive dataset mixing and calls for future work on principled gradient-level corrections for LLMs. † † affiliation: Meta AI † † affiliation: Meta Superintelligence Labs † † affiliation: Cornell Tech † † affiliation: Columbia University † † affiliation: Technion † † affiliation: Tel Aviv University † † contribution: Work done at Meta † † date: August 10, 2026 † † correspondence: Runzhe Wu at [email protected] 1 Introduction Large language models (LLMs) are increasingly trained to master multiple tasks simultaneously, from language understanding and summarization to code generation and math problem solving. Such multi-task learning enables models to generalize across domains and learn more efficiently than by training separate models per task ( Qi et al. 2024 ; Brief et al. 2024 ) . A standard approach to multi-task post-training is to merge datasets from different tasks and train on them jointly ( Team et al. 2023 ; Wang et al. 2023a ; Wang et al. 2023b ; Hu et al. 2024 ; Zhu et al. 2025 ) . While simple, this strategy effectively mixes gradients across tasks and implicitly requires gradient magnitudes to be similar across tasks. When some tasks generate much larger gradients, the optimization becomes dominated by them. Such imbalance has been noted in the broader field such as vision ( Chen et al. 2018 ) but is less explored in the LLM literature ( Appendix A ). In this paper, we show that such gradient imbalance arises prominently in the RL post-training of multi-task LLMs ( Figure 1 ). The consequence of this imbalance is detrimental: (1) when certain tasks produce much larger gradients, the gradients aggregated from all tasks are dominated by them, which biases updates toward large-gradient tasks and reduces progress on the rest; and (2) from an optimization perspective, large-gradient tasks are effectively trained as if their learning rates were set to be larger than those of small-gradient tasks. Therefore, either large-gradient tasks are optimized too aggressively, or small-gradient tasks are under-optimized. Crucially, such gradient imbalances would only be reasonable if larger gradients correspond to greater learning gains; i.e., large-gradient tasks should dominate optimization only when they indeed yield larger improvements on their respective objectives (in our case, training rewards). However, we find this hypothesis does not hold: tasks with much larger gradients can exhibit similar or even lower learning gains than those with much smaller gradients. To further examine the correlation between gradient magnitude and learning gains, we implement gradient-proportional sampling that allocates more training to large-gradient tasks to see if it brings any advantage—the rationale is that if gradient magnitude truly reflects learning gains, then training more frequently on large-gradient tasks should improve overall average performance by prioritizing tasks with greater gains while sacrificing those with smaller ones ( Chen et al. 2025 ; Wang et al. 2025 ) . However, we found this approach shows no advantages and can perform even worse in our experiments, proving that the gradient signal across tasks is not merely uncorrelated with learning gains but misleading when used for training signals. In summary, we find that gradient imbalance is not caused by imbalanced learning gains and can thus harm multi-task learning by biasing updates. To probe the source of this imbalance, we examined some training statistics as potential explanations, such as training rewards and advantage functions, but we found none could account for it. It suggests that the gradient imbalance does not depend on these training statistics but arises from inherent differences between tasks. This highlights the need for future methods that explicitly correct it in training. 2 Imbalanced Gradients Figure 1: Gradient imbalance and its misalignment with learning gains. Top: Qwen-7B trained on four heterogeneous tasks. The left panel shows squared gradient norms, with Code dominating throughout training. However, the middle panel reveals that Code achieves the lowest learning gains—defined as the increase in training reward ( Section 3.1 ). The right panel reports training rewards for reference. Bottom: Qwen-7B trained on three math tasks. Arithmetic exhibits up to 33\times larger squared gradients but major learning gains vanish after early training. Together, these results indicate an implicit training bias due to imbalanced gradients which cannot be attributed to learning gains. Plots of other models are provided in Section D.2 . In this section, we provide the empirical evidence of gradient imbalance in multi-task RL post-training. We study different models: instruction-tuned versions of Qwen2.5-3B and 7B ( Qwen et al. 2024 ) and Llama-3.2-3B ( Dubey et al. 2024 ) . We consider RLVR ( Guo et al. 2025 ) and use GRPO ( Shao et al. 2024 ) as the RL algorithm. Additional details of the implementation and the results are reported in Appendices C and D . We consider two multi-task settings: (1) four different tasks across multiple domains and (2) three tasks within the math domain but with varying difficulty. They are described below. Multi-domain tasks. We focus on four tasks: (1) Code , involving code generation in the style of programming contests ( Liu and Zhang 2025 ) ; (2) Countdown , where the model needs to construct a target number from given inputs using basic arithmetic operators ( Pan et al. 2025 ) ; (3) MATH , focused on mathematical problem solving ( Hendrycks et al. 2021 ) ; and (4) FinQA , financial reasoning over tabular data and text ( Chen et al. 2021 ) . This setup focuses on examining the gradient behaviors across heterogeneous tasks. Single-domain (math) tasks. We consider three datasets related to mathematics. DeepScaleR ( Luo et al. 2025 ) and MATH ( Hendrycks et al. 2021 ) contain challenging math problems (with DeepScaleR being harder overall). Arithmetic ( Brown et al. 2020 ) consists of basic arithmetic problems and is easier. This setup allows us to examine gradient behavior within a single domain while varying task difficulty and problem styles. We conduct experiments for both multi-task settings described above. During training, batches are constructed by sampling from all tasks with equal probability—importantly, we sample with equal probability rather than simply mixing the datasets from all tasks because the latter leads to a natural bias due to different dataset sizes across tasks. To study gradient imbalance,