Uniform Loss vs. Specialized Optimization: A Comparative Analysis in Multi-Task Learning

Gabriel S. Gama, Valdir Grassi Jr · 2025

Read Uniform Loss vs. Specialized Optimization: A Comparative Analysis in Multi-Task Learning by Gabriel S. Gama, Valdir Grassi Jr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10347.txt.

Introduction

Gabriel Gama and Valdir Grassi compare training schemes for multi-task learning in this two thousand twenty-five paper. When one model learns several tasks at once, the tasks can fight each other. Their gradients may clash or differ in size. Specialized optimizers were built to balance this fight. Recent critics claim plain equal weighting does just as well, if tuned with care. The authors set out to settle the dispute with broad tests.

They ran the latest specialized methods on harder multi-task problems than past studies used. The specialists did perform well against plain uniform loss. Yet fixed well-chosen weights also stayed competitive. The paper explains when uniform loss keeps pace and when it falls behind. Code is shared for others to check the claims.

Catalog record

CREATORGabriel S. Gama, Valdir Grassi Jr
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-10347

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Uniform Loss vs. Specialized Optimization: A Comparative Analysis in Multi-Task Learning Gabriel S. Gama, Valdir Grassi Jr Abstract Specialized Multi-Task Optimizers (SMTOs) balance task learning in Multi-Task Learning by addressing issues like conflicting gradients and differing gradient norms, which hinder equal-weighted task training. However, recent critiques suggest that equally weighted tasks can achieve competitive results compared to SMTOs, arguing that previous SMTO results were influenced by poor hyperparameter optimization and lack of regularization. In this work, we evaluate these claims through an extensive empirical evaluation of SMTOs, including some of the latest methods, on more complex multi-task problems to clarify this behavior. More specifically, we start our analysis by evaluating all SMTOs on a simple MNIST problem to identify the promising optimizers and assess them further on progressively more complex multi-task problems. Our findings indicate that SMTOs perform well compared to uniform loss and that fixed weights can achieve competitive performance compared to SMTOs. Furthermore, we demonstrate why uniform loss performs similarly to SMTOs in some instances. The source code is available at https://github.com/Gabriel-SGama/UnitScal_vs_SMTOs . 1 Introduction When tackling complex real-world challenges in machine learning, one must contemplate the utility of Multi-Task Learning (MTL) [ 1 ] . Scenarios such as scene understanding, multilingual translation, and predicting chemical properties require multiple outputs, possibly making them promising candidates for MTL approaches due to their inherent inductive bias [ 2 ] . Classical methods approach this problem by utilizing one model for each task, but most MTL scenarios have the potential to share information across tasks. For example, autonomous driving relies on multiple pieces of information extracted from the same visual input, such as semantic segmentation, object detection, instance segmentation, and others. Therefore, sharing information across those tasks is desirable to increase performance. To efficiently share feature information across tasks, various approaches have been proposed, including model architecture designs [ 3 , 4 , 5 ] , task grouping strategies [ 6 , 7 ] , Meta-Learning techniques [ 8 ] , and the focus of this work, Specialized Multi-Task Optimizers (SMTOs) [ 9 , 10 ] . SMTOs are implemented together with a shared encoder architecture, where one common encoder outputs a feature space that is shared with the specific decoders used for each task [ 9 , 10 ] . Consequently, this reduces the computational cost when compared to the individual models approach. This unified model can be optimized by minimizing the sum of the losses, known as uniform loss. However, the simultaneous learning of multiple tasks introduces challenges such as conflicting directions and significant gradient norm disparities [ 11 , 12 ] . SMTOs aim to address these challenges by applying mathematical tools to jointly and equally optimize all objectives. SMTOs can be classified into two types: gradient-based and loss-based. The former compute the gradients of each task and combine them to find a suitable optimization direction [ 9 , 13 , 14 , 15 , 11 , 12 , 10 , 16 ] . This process adds complexity and computational cost to the learning problem as multiple backpropagation steps are needed. The loss-based methods only take into consideration the loss values [ 17 , 18 , 19 , 20 , 21 ] , reaching computational times comparable to those of the uniform loss [ 22 , 21 ] . One notable issue in the SMTO field is the lack of a standardized evaluation framework and procedure, leading to concerns about reproducibility [ 23 ] . Understandably, some works have begun questioning the efficacy of SMTOs [ 24 , 23 ] . These critiques attribute reported gains in SMTO papers to sub-optimal hyperparameter tuning or insufficient regularization. They provide various experiments, including natural language processing (NLP), computer vision, and reinforcement learning, as well as a theoretical hypothesis on why SMTOs apply a regularization effect and a comparison to a properly optimized baseline. In this context, we propose to address these concerns by hypothesizing that proper hyperparameter optimization is crucial for fair performance assessment of SMTOs. Nevertheless, we also emphasize the significance of considering the inherent complexity of the MTL balancing problem. Our contributions include: • A comprehensive evaluation of current SMTOs throughout Section 3 , including recent ones that have not been covered in previous critiques, using the MNIST dataset [ 25 ] as a starting point to identify promising SMTOs, followed by other commonly used datasets such as Cityscapes [ 26 ] and Quantum Chemistry and Molecular Physics (QM9) [ 27 ] . • An investigation into why equally weighted tasks can match the results of SMTOs. More specifically, the two similar tasks scenario in Section 3.2 , and a complexity analysis in Section 3.3 . • A re-evaluation of the hypothesis brought up by [ 23 ] that fixed weights can match the same results achieved by SMTOs in Section 3.5 . 2 Related Work In recent years, MTL and SMTOs have witnessed substantial research activity. Notable works, such as [ 17 , 9 ] , have garnered interest in MTL due to their performance improvements compared to the uniform loss and single task baselines. This surge has also given rise to numerous variations and novel SMTO methods, each improving on or proposing solutions to unaddressed concerns of early SMTOs [ 14 , 28 , 12 , 16 ] . However, despite the growing body of research, the SMTO field faces a significant challenge: the absence of a formal benchmark. Consequently, reported results may vary between articles [ 9 , 23 , 10 ] , raising concerns about the improvements. Specifically, recent critiques have revealed potential issues, suggesting that the reported performance gains from SMTOs stem from limited regularization [ 24 ] and under-optimized baseline hyperparameters [ 23 ] . Kurin et al. [24] show that all evaluated SMTOs are linked to a larger solution space, under-optimization or stochastic behavior, all of which induce a regularization effect [ 29 , 30 , 31 ] . They also evaluated their hypothesis by adding proper regularization such as weight decay, batch normalization [ 32 ] , dropout [ 33 ] , and early stopping to the uniform loss, referenced as Unitary Scalarization (Unit. Scal.) in this paper, and compared it to SMTOs. After the changes, they achieved competitive results to SMTOs at a lower computational cost. Similarly, Xin et al. [23] argue that reported SMTO gains may be attributed to poor hyperparameter optimization, with particular interest in the impact of learning rates compared to random seeds. Their work demonstrates that exploring multiple fixed weight values can construct a Pareto curve, and SMTOs can only obtain one of the possible solutions. Due to computational restrictions, the discussion regarding fixed weights will be partially addressed as, to the best of our knowledge, there is no method to predict those fixed weights accurately, and grid search becomes impractical for a higher number of tasks. Our work distinguishes itself in two key ways. Firstly, we consider most of the recent SMTOs in our evaluation [ 28 , 12 , 10 , 18 , 16 , 20 ] . Secondly, we adopt some variations to the commonly used problems to introduce more complexity, as detailed in Section 3 . Our objective is to demonstrate

Read the complete text