Multi-task learning on partially labeled datasets via invariant/equivariant semi-supervised learning

Miquel Martí i Rabadán [email protected], Alessandro Pieropan [email protected], Hossein Azizpour [email protected], Atsuto Maki [email protected] · 2026

Read Multi-task learning on partially labeled datasets via invariant/equivariant semi-supervised learning by Miquel Martí i Rabadán [email protected], Alessandro Pieropan [email protected], Hossein Azizpour [email protected], Atsuto Maki [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17624.txt.

Introduction

Miquel Martí i Rabadán and colleagues test invariant and equivariant semi-supervised learning for multi-task models on partial labels. They use FixMatch for the invariant case and its extension Dense FixMatch, tested on Cityscapes and BDD100K for detection and scene parsing. Varying size and overlap of per-task labeled subsets, both approaches beat plain baselines in most cases, with the largest gains when few labels exist and better results for the latter approach. Both approaches thus look promising for multi-task learning from limited data.

Catalog record

CREATORMiquel Martí i Rabadán [email protected], Alessandro Pieropan [email protected], Hossein Azizpour [email protected], Atsuto Maki [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59506
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17624

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-task learning on partially labeled datasets via invariant/equivariant semi-supervised learning Miquel Martí i Rabadán [email protected], Alessandro Pieropan [email protected], Hossein Azizpour [email protected], Atsuto Maki [email protected] Abstract We investigate the potential of invariant and equivariant semi-supervised learning for addressing the challenges of training multi-task models on partially labeled datasets with differently structured output tasks. Specifically, we use the popular FixMatch method for invariant semi-supervised learning and its equivariant extension Dense FixMatch. We evaluate their performance on the Cityscapes and BDD100K datasets in the context of the prevalent object detection and semantic segmentation tasks in computer vision. We consider varying sizes of the subsets annotated for each task and different overlaps among them. Our results for both invariant and equivariant semi-supervised learning outperform supervised baselines in most situations, with the most significant improvements observed when fewer labeled samples are available for a task and generally better results for the latter approach. Our study suggests that invariant/equivariant learning is a promising general direction for multi-task learning from limited labeled data. 1 Introduction Multi-task learning has emerged as a powerful paradigm in deep learning, with the potential to enable efficient and effective solutions to a range of real-world problems. With the explosion of embedded devices and the growing demand for real-time applications, the ability to solve multiple tasks simultaneously has become a critical requirement. By sharing a common backbone among tasks, multi-task models reduce total system requirements regarding inference-time memory footprint, computation cost, and storage needed for model weights ( 41 ) . Additionally, simultaneously training multiple tasks can improve performance by leveraging task relations or regularizing each other ( 5 ; 4 ; 21 ; 51 ) . However, multi-task deep learning brings its own challenges, including designing effective architectures and optimizing for multiple objectives, among others ( 41 ) . This work focuses on the challenges related to the partial annotations available for different tasks in multi-task datasets. Firstly, the number of samples annotated for different tasks can vary greatly, typically depending on the annotation cost of each task. Dense tasks such as semantic segmentation tend to have a per-sample annotation cost orders of magnitude higher than image-level labels ( 2 ; 9 ) . Other factors, such as the expertise required by the annotators, can also play an important role. For example, annotations from qualified physicians can cost significantly more in medical imaging. In contrast, certain tasks can be annotated at no additional cost, as the loss function can be defined in a self-supervised manner or via other data modalities collected simultaneously. For instance, depth estimation can be performed through stereo image pairs or video sequences ( 12 ; 13 ) . As a result, these different costs lead to imbalances between labeled subsets for each task, limited-sized datasets, or high costs to fully annotate all samples. Furthermore, machine learning projects are iterative, involving multiple cycles of data collection and labeling, training, and testing. At the same time, priorities and requirements often change, and it is common to have different teams working on different tasks. These accentuate the challenge of learning from such data by making it more likely to have only partially overlapping subsets annotated for each task or possibly fully disjoint datasets. Learning from such partially labeled datasets using a fully-supervised approach significantly limits the number of samples available for training. One simple solution is to cancel the losses of tasks for samples with no corresponding annotations ( 21 ) . This approach still under-utilizes data for the less-frequently annotated tasks, leading to suboptimal performance ( 32 ) . Instead, our goal is to involve all available samples in learning all tasks, hopefully aiding the generalization of all tasks. Another way to learn from partially labeled multi-task data is by casting it as multiple semi-supervised learning problems solved simultaneously, where from the perspective of each task, there is a set of labeled samples and another set of unlabeled samples. Using semi-supervised learning methods for each task allows the involvement of all data points in the learning of all tasks ( 32 ; 24 ; 20 ) . We put special emphasis on partially labeled multi-task problems with tasks exhibiting output structures that can be equivariant to certain transformations of the inputs, i.e. equivariant tasks. Typical examples are structured tasks such as the computer vision tasks of object detection and semantic segmentation, which produce outputs equivariant to affine transformations of the input such as rotations or translations, but with different structures: segmentation maps vs. bounding boxes. To this end, we propose invariant/equivariant learning as a task-generic semi-supervised learning paradigm that can be adopted for multiple tasks with different output structures simultaneously. Invariant or equivariant learning enforces invariance or equivariance to certain input transformations respectively, a known property of the task, and uses it as a learning objective that can be defined on unlabeled data. We use FixMatch ( 38 ) as the invariant learning approach, using only photometric transformations to allow its use with equivariant tasks, which we refer to as FixMatch*. Dense FixMatch ( 30 ) , which extends FixMatch by using equivariance to affine transformations as a learning objective on unlabeled data, as well as invariance to photometric transformations, is referred throughout the paper as equivariant learning with a slight abuse of language since it does enforce both invariance and equivariance to photometric and affine transformations respectively. Both methods can be applied to multi-task learning on partially labeled datasets in a broader setting where tasks do not share the same output structure, such as semantic segmentation and object detection tasks, while using the core components of modern, state-of-the-art semi-supervised methods for different tasks. We study the use of invariant/equivariant learning to address the challenges presented by multi-task learning on partially labeled datasets with the Cityscapes ( 9 ) and BDD100K ( 50 ) computer vision datasets using partial annotations for semantic segmentation and object detection tasks. Our main contributions are: • We conduct comprehensive experiments representing real-world multi-task learning scenarios by varying annotation amounts for each task, annotation overlaps, and dataset sizes. • We find that both invariant learning with FixMatch* and equivariant learning with Dense FixMatch generally outperform supervised baselines, more notably for tasks with limited annotations, while the latter approach gives better results for most of the cases we studied. In light of our findings, we view invariant/equivariant learning as a promising general approach that could be further enhanced when paired with task-specific methods, or other forms of self-supervision when possible. 2 Related work Semi-supervised learning aims to employ unlabeled data alongside smaller amounts of labeled data to improve generalization performance ( 49 ) . Self-training ( 23 ; 46 ) and consistency regularization ( 22 ; 40 ; 31 ) have been the leading approaches to semi-supervised deep learning. Self-training involves using the model’s own predictions on unlabeled data to generate additional labeled data, then used to train the model further. Consistency regularization enforces consistency to small perturbations by using unlabeled samples as targets and minimizing the discrepancy between the model’s predictions on different perturbations of the same sample. State-of-the-art solutions combine both and use transformations on the input data as the perturbations ( 45 ; 3 ) . In particular, FixMatch ( 38 ) enforces

Read the complete text