Exploring Transferability of Self-Supervised Learning by Task Conflict Calibration
Huijie Guo, Jingyao Wang, Peizheng Guo, Xingchen Shen, Changwen Zheng, Wenwen Qiang · 2025
Read Exploring Transferability of Self-Supervised Learning by Task Conflict Calibration by Huijie Guo, Jingyao Wang, Peizheng Guo, Xingchen Shen, Changwen Zheng, Wenwen Qiang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13787.txt.
Introduction
Huijie Guo and 5 co-authors probe what makes self-supervised representations transfer to new tasks, and how to improve that transfer. Transferability means features learned on one objective still serve another, but task conflict during training can quietly damage it. Their approach builds several self-supervised tasks inside each training batch so transfer can be modeled directly.
Evidence and causal analysis show task-level detail helps, yet conflict still drags results down. Their Task Conflict Calibration method extracts causal factors for each task, weights samples per task, and recalibrates representations inside a two-stage optimization loop. Across multiple downstream tasks, calibrated models consistently transferred better than standard self-supervised training.
Catalog record
| CREATOR | Huijie Guo, Jingyao Wang, Peizheng Guo, Xingchen Shen, Changwen Zheng, Wenwen Qiang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-13787
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring Transferability of Self-Supervised Learning by Task Conflict Calibration Huijie Guo, Jingyao Wang, Peizheng Guo, Xingchen Shen, Changwen Zheng, Wenwen Qiang Abstract In this paper, we explore the transferability of SSL by addressing two central questions: (i) what is the representation transferability of SSL, and (ii) how can we effectively model this transferability? Transferability is defined as the ability of a representation learned from one task to support the objective of another. Inspired by the meta-learning paradigm, we construct multiple SSL tasks within each training batch to support explicitly modeling transferability. Based on empirical evidence and causal analysis, we find that although introducing task-level information improves transferability, it is still hindered by task conflict. To address this issue, we propose a Task Conflict Calibration (TC 2 ) method to alleviate the impact of task conflict. Specifically, it first splits batches to create multiple SSL tasks, infusing task-level information. Next, it uses a factor extraction network to produce causal generative factors for all tasks and a weight extraction network to assign dedicated weights to each sample, employing data reconstruction, orthogonality, and sparsity to ensure effectiveness. Finally, TC 2 calibrates sample representations during SSL training and integrates into the pipeline via a two-stage bi-level optimization framework to boost the transferability of learned representations. Experimental results on multiple downstream tasks demonstrate that our method consistently improves the transferability of SSL models. Code — https://github.com/PaulGHJ/TC2 Introduction Traditional unsupervised learning lacks ground-truth labels and often performs worse than supervised learning. However, supervised methods heavily rely on costly manual annotations and still face generalization issues ( 6 ; 36 ) . As a compelling alternative, Self-Supervised Learning (SSL) has shown strong performance in image classification and object recognition ( 47 ; 51 ; 34 ) , sometimes even surpassing supervised methods. Existing self-supervised learning (SSL) methods primarily focus on designing reliable supervisory signals or introducing inductive biases to guide the model in learning effective feature representations. For example, discriminative SSL (D-SSL) methods construct positive pairs by generating different augmented views of the same sample and treat other samples as negatives ( 9 ; 25 ; 7 ) . Generative SSL (G-SSL) methods mask part of the input and train the model to reconstruct the masked regions, using the reconstruction error as the supervisory signal ( 24 ; 3 ) . However, these methods often overlook a critical question: how do such learning paradigms ensure the transferability of learned features? This question is not easy to answer and helps explain why these methods often perform poorly on out-of-distribution (OOD) data or transfer tasks. Additionally, this limits the theoretical insights and methodological advances needed to improve their transfer generalization capabilities. In this paper, we focus on two key questions: (1) What is representation transferability? (2) How can it be effectively modeled? For the first question, we take an intuitive perspective: if a representation has good transferability, it should perform well across various downstream tasks. Based on this intuition, we define task-level transferability as the ability of a representation learned from one task to support the objective of another ( 66 ) . To address the second question, we consider meta-learning as an established approach to modeling transferability. Prior works ( 40 ) suggest that SSL can be regarded as a special case of meta-learning. The main difference lies in task organization within each mini-batch and learning mechanism. Meta-learning typically involves multiple tasks, whereas SSL is generally considered to involve only a single task. Moreover, meta-learning is optimized in a bi-level manner. It achieves strong transferability through multi-task bi-level training, which implicitly models a distribution over tasks. This allows meta-learning to generalize well to unseen tasks. Motivated by this, we propose enhancing the transferability of SSL by introducing a meta-learning mechanism. Specifically, we construct multiple tasks within each mini-batch during training and train SSL models in a bi-level manner. (a) (b) (c) Figure 1: Effect of task-level information on CIFAR-100. (a) shows the effect of SimCLR trained on ImageNet and tested on CIFAR-100 before and after introducing task-level information in five runs. (b) and (c) show the training process corresponding to the worst and best rounds in five evaluations, visualizing the gradient similarity between tasks. We further explore the effectiveness of the above meta-learning-like SSL method on transferability. In the experiment, we uniformly split each training batch into two subsets, which serve as two independent SSL tasks. Following the standard SSL training framework, each subset generates two augmented views via random data augmentation for training. We use SimCLR as the baseline, training on ImageNet-100 and testing on other datasets. First, we evaluate the Top-1 accuracy without task construction. Next, we assess performance with the multi-task learning mechanism through five independent training rounds and calculate the improvement for each round. From Figure 1(a) , the results demonstrate that incorporating task-level information significantly improves model performance, but the improvement is not consistent, with a nearly 3% performance gap between the best and worst outcomes, and some cases even showing negative gains. Further analysis reveals that gradient conflict between tasks during training significantly affects transferability, as shown in Figure 1(b) and 1(c) . Therefore, when introducing task-level information enhances transferability, inter-task gradient conflict remains an important issue. To better understand the underlying causes of task conflict and its impact on transferability, we analyze it through both data generation and causal inference. Specifically, we construct a Structural Causal Model (SCM) to illustrate these relationships (as shown in Figure 2(a) ), where Y_{i} , Y_{j} are task labels, F_{i,j}^{s} is the shared generative factors, and F_{i}^{u} , F_{j}^{u} are task-specific factors. In meta-learning-like SSL, the model estimates generative factors from multiple tasks and predicts labels accordingly. Due to the shared training across tasks, the extracted factors inevitably mix semantics from all tasks(Figure 2(b) ). As shown in Theorem 1 , if task-specific factors are not correctly identified, these factors from other tasks act as confounders, hindering target task learning—this phenomenon is termed task conflict. It degrades per-task learning and limits the effectiveness of modeling task distributions, explaining the inconsistent transfer performance observed in meta-learning-like SSL. Moreover, these generative factors can be seen as semantic vectors composing the feature space. Since different tasks may involve distinct semantics, task conflict implies that target features may be contaminated by irrelevant semantics. Thus, enhancing SSL transferability requires calibrating each sample’s feature representation to retain only task-relevant semantics within the meta-learning-like framework. Based on the above analysis, we propose a new method, called Task Conflict Calibration (TC 2 ), to alleviate task conflict when explicitly modeling the transferability for SSL methods. First, TC 2 constructs multiple SSL tasks through batch splitting, which introduces task-level information into the model. Second, TC 2 learns a factor extraction network f_{v} to generate causal generative factors for all tasks and a weight extraction network f_{w} to generate a weight of a single sample of each task. We leverage techniques such as data reconstruction, orthogonality, and sparsity to ensure that the learned f_{v} can effectively generate causal generative factors suitable for multiple tasks, while the learned function f_{w} produces a weight matrix of semantic vectors that are specifically relevant to the target task. Third, based on f_{v} and f_{w} , we formulate the final