Delve into the Applicability of Advanced Optimizers for Multi-Task Learning
Zhipeng Zhou, Linxiao Cao, Pengcheng Wu, Peilin Zhao, Chunyan Miao · 2026
Read Delve into the Applicability of Advanced Optimizers for Multi-Task Learning by Zhipeng Zhou, Linxiao Cao, Pengcheng Wu, Peilin Zhao, Chunyan Miao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08939.txt.
Introduction
Zhou and four colleagues ask why powerful optimizers weaken multi-task learning, in which one model learns several jobs together. They show that gradients taken at each moment play only a small part in the parameter updates, so schemes meant to resolve clashes between tasks fall short. They also observe that the Muon optimizer already behaves like a multi-task learner. Their APT framework pairs adaptive momentum with a light method that preserves direction for Muon, and trials on four task sets lift existing approaches.
Catalog record
| CREATOR | Zhipeng Zhou, Linxiao Cao, Pengcheng Wu, Peilin Zhao, Chunyan Miao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35584 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.08939
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Delve into the Applicability of Advanced Optimizers for Multi-Task Learning Zhipeng Zhou, Linxiao Cao, Pengcheng Wu, Peilin Zhao, Chunyan Miao Abstract Multi-Task Learning (MTL) is a foundational machine learning problem that has seen extensive development over the past decade. Recently, various optimization-based MTL approaches have been proposed to learn multiple tasks simultaneously by altering the optimization trajectory. Although these methods strive to de-conflict and re-balance tasks, we empirically identify that their effectiveness is often undermined by an overlooked factor when employing advanced optimizers: the instant-derived gradients play only a marginal role in the actual parameter updates. This discrepancy prevents MTL frameworks from fully releasing its power on learning dynamics. Furthermore, we observe that Muon—a recently emerged advanced optimizer—inherently functions as a multi-task learner, which underscores the critical importance of the gradients used for its orthogonalization. To address these issues, we propose APT (Applicability of advanced oPTimizers), a framework featuring a simple adaptive momentum mechanism designed to balance the strengths between advanced optimizers and MTL. Additionally, we introduce a light direction preservation method to facilitate Muon’s orthogonalization. Extensive experiments across four mainstream MTL datasets demonstrate that APT consistently augments existing MTL approaches, yielding substantial performance improvements. 1 Introduction Traditional machine learning typically relies on customized models tailored with task-specific knowledge. However, in modern real-world scenarios, the requirement to handle multiple tasks simultaneously has made the development of Multi-Task Learning (MTL) models increasingly essential. MTL is a mature field with diverse methodologies, including architecture-based approaches 33 ; 6 , optimization-based methods 13 ; 11 , and task relationship modeling 30 ; 5 . This paper focuses exclusively on optimization-based MTL, which assumes a fixed architecture comprising a shared backbone and task-specific branches 16 . These methods aim to design effective re-weighting strategies for task gradients or losses. Recent benchmarks 2 ; 35 demonstrate that advanced optimization-based MTL achieves competitive performance, sometimes even surpassing Single-Task Learning (STL) upper bounds. Nevertheless, we contend that the effectiveness of these approaches is fundamentally undermined by the use of Exponential Moving Average (EMA) in modern optimizers, e.g., Adam series 9 ; 18 , and Muon 7 , etc. As illustrated in Figure 1 , even when a de-conflicted direction g_{mtl} is derived, the EMA mechanism causes the momentum direction g_{mom} to dominate, resulting in a final update g_{update} that remains conflicted. This suggests that current MTL mechanisms do not achieve the instant de-conflicting intended; instead, they amortize MTL effects over time. This claim is further supported by our empirical observations in Section 3.2 . While EMA is beneficial for noise reduction and stability, its use creates a critical trade-off between instant and amortized de-conflicting ( Challenge 1 ). In this work, we focus on Adam, AdamW, and Muon as representative advanced optimizers for our study. Figure 1: Illustration of the impact of EMA in advanced optimizer for MTL. Here we adopt g_{update}=0.9*g_{mom}+0.1*g_{mtl} as the example. Furthermore, Muon—a recently proposed advanced optimizer—requires additional consideration. We show that Muon can be interpreted as an implicit MTL learner, as its orthogonalization operation tends to amplify components shared among task gradients. This property places stricter requirements on the quality of the aggregated input gradient, giving rise to a second challenge ( Challenge 2 ). To address these challenges, we propose a simple adaptive momentum strategy that dynamically balances instant and amortized de-conflicting based on local curvature. To resolve Challenge 2 , we additionally introduce a light direction preservation approach designed to facilitate Muon’s orthogonalization process. In a nutshell, our contributions can be summarized into three-fold: • We reveal that mainstream optimization-based MTL approaches primarily achieve amortized rather than instant de-conflicting, which may deviate from common expectations and partially explains their limitations in practice. • To jointly benefit from the denoising and smoothing effects of EMA and the de-conflicting capability of MTL, we propose an adaptive momentum strategy guided by curvature information. In addition, we introduce a lightweight direction preservation (LDP) approach to derive a more favorable input gradient for the Muon optimizer. • Extensive experiments across multiple mainstream MTL benchmarks have demonstrated that our method successfully augments mainstream MTL approaches, achieving a new competitive state-of-the-art (SOTA) performance. 2 Related Work 2.1 Optimization-based Multi-Task Learning In optimization-based MTL, a wide range of methods have been proposed 36 ; 34 , which can be broadly categorized into gradient-based and loss-based approaches. Gradient-based methods operate directly on task gradients with respect to shared parameters and aim to resolve conflicts through gradient manipulation. For example, MGDA 26 seeks a convex combination of task gradients with minimum norm. PCGrad 32 addresses gradient conflicts by projecting conflicting gradients onto orthogonal subspaces. CAGrad 13 balances minimum-norm optimization with convergence guarantees. Align-MTL 27 performs singular value decomposition (SVD) on task gradients and re-weights gradient components in the resulting subspace. Nash-MTL 21 and FairGrad 2 both formulate MTL as a fair allocation problem across tasks, with Nash-MTL being a special case under certain settings. COST 35 tackles gradient conflicts by seeking alternative update directions that lie on the same loss level set. In contrast, loss-based methods adjust task weights based solely on task losses computed from model outputs, without explicitly using task gradients. As a result, they are typically more lightweight but often underperform gradient-based approaches due to the lack of gradient-level information. Early loss-based methods include RLW 10 , which samples task weights from a random distribution, and Uncertainty Weighting 8 , which derives weights based on task-specific predictive uncertainty. More recent loss-based approaches are commonly formulated within a bi-level optimization framework, differing mainly in their lower-level task-weight update strategies. For instance, FAMO 11 amortizes the MGDA objective over time using a first-order approximation. Go4Align 28 introduces dynamic task grouping in the lower-level optimization. LDC-MTL 31 employs a routing network to regulate loss discrepancies during training. Table 1: Analysis on \beta_{1} across various MTL approaches and advanced optimizers on CityScapes . AdamW is set with weight decay as 1.e-4 to distinguish with Adam. All results are average on 3 random seeds. The best scores are provided in gray . \beta_{1} MGDA CAGrad Adam AdamW Muon Adam AdamW Muon default 7.29 \pm 4.61 6.41 \pm 5.71 3.37 \pm 4.02 12.38 \pm 1.49 10.57 \pm 1.80 -2.37 \pm 0.54 0.6 9.21 \pm 4.12 5.82 \pm 3.37 4.14 \pm 1.10 9.78 \pm 1.14 10.88 \pm 1.54 1.95 \pm 0.90 0.2 9.25 \pm 3.75 5.17 \pm 3.24 19.39 \pm 6.64 10.78 \pm 3.03 10.07 \pm 0.96 81.12 \pm 9.13 0.0 3.52 \pm 1.50 10.23 \pm 4.68 33.35 \pm 6.61 9.11 \pm 0.97 10.08 \pm 2.58 120.78 \pm 36.70 \beta_{1} Nash-MTL FairGrad Adam AdamW Muon Adam AdamW Muon default 24.87 \pm 1.57 23.42 \pm 1.61 -4.22 \pm 0.66 4.23 \pm 3.20 4.04 \pm 0.66 6.04 \pm 1.16 0.6 22.59 \pm 1.39 21.62 \pm 1.89 3.15 \pm 2.97 4.81 \pm 2.77 3.53 \pm 4.19 1.63 \pm 3.05 0.2 20.45 \pm 1.68 19.05 \pm 2.30 77.04 \pm 27.68 2.86 \pm 1.13 2.94 \pm 0.88 44.29 \pm 25.49 0.0 20.68 \pm 2.04 19.68 \pm 0.99 92.93 \pm 5.26 2.38 \pm 0.90 3.26 \pm 1.22 55.87 \pm 28.48 2.2 Advanced Optimizers Advanced optimizers in deep learning are dominated by the Adam family, which builds on exponential moving averages of first- and second-order moments to provide per-parameter adaptive steps