Mitigating Negative Transfer via Reducing Environmental Disagreement PubID: pubid: 0000–0000/00$00.00 © 20xx IEEE
Hui Sun, Zheng Xie, Hao-Yuan He, and Ming Li · 2025
Read Mitigating Negative Transfer via Reducing Environmental Disagreement PubID: pubid: 0000–0000/00$00.00 © 20xx IEEE by Hui Sun, Zheng Xie, Hao-Yuan He, and Ming Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24044.txt.
Introduction
Hui Sun and three co-authors study how a model trained in one setting can fail when used in a new setting with no labels. This failure is called negative transfer, where the move makes results worse instead of better. They trace the problem to features that are not true causes but shift with the environment.
When a model leans too much on those shifting features, its choices clash across settings and hurt transfer. Their method, RED, splits causal features from environmental ones and cuts that clash. Tests support the idea and show better results when the gap between source and target settings is large.
Catalog record
| CREATOR | Hui Sun, Zheng Xie, Hao-Yuan He, and Ming Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-24044
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mitigating Negative Transfer via Reducing Environmental Disagreement PubID: pubid: 0000–0000/00$00.00 © 20xx IEEE Hui Sun, Zheng Xie, Hao-Yuan He, and Ming Li Abstract Unsupervised Domain Adaptation (UDA) focuses on transferring knowledge from a labeled source domain to an unlabeled target domain, addressing the challenge of domain shift . Significant domain shifts hinder effective knowledge transfer, leading to negative transfer and deteriorating model performance. Therefore, mitigating negative transfer is essential. This study revisits negative transfer through the lens of causally disentangled learning, emphasizing cross-domain discriminative disagreement on non-causal environmental features as a critical factor. Our theoretical analysis reveals that overreliance on non-causal environmental features as the environment evolves can cause discriminative disagreements (termed environmental disagreement ), thereby resulting in negative transfer. To address this, we propose Reducing Environmental Disagreement (RED), which disentangles each sample into domain-invariant causal features and domain-specific non-causal environmental features via adversarially training domain-specific environmental feature extractors in the opposite domains. Subsequently, RED estimates and reduces environmental disagreement based on domain-specific non-causal environmental features. Experimental results confirm that RED effectively mitigates negative transfer and achieves state-of-the-art performance. I Introduction Transfer learning is a cornerstone of modern machine learning and plays a crucial role in developing robust artificial intelligence (AI) systems in open and dynamic environments [ 1 , 2 ] . Transfer learning enables AI systems to generalize to new tasks by leveraging knowledge gained from previous tasks, thereby avoiding the time- and resource-intensive process of collecting supervised data and training models from scratch. One of the most primary challenge in transferring is the distribution shift between data from new tasks and previous tasks. Specifically, the prominent subfield of transfer learning, unsupervised domain adaptation (UDA), focuses on transferring knowledge from a labeled source domain to an unlabeled target domain [ 3 , 4 , 5 ] . Modern UDA methods harness the power of deep neural networks to learn domain-invariant feature representations, aligning cross-domain distributions in the representation space. These features serve as a bridge between the source and target domains, facilitating knowledge transfer across domains. However, in the presence of substantial distribution shift, bridging and transferring knowledge between domains become challenging and may harm the model’s performance, a phenomenon widely recognized as negative transfer [ 6 , 7 ] . (a) Classic Disentanglement (b) Causal Disentanglement Fig. 1: Disentanglement comparison: Fig. 1a illustrates feature disentanglement in classic UDA, which assumes absolute independence between domain-specific and class-specific features. Fig. 1b presents a more general perspective on feature disentanglement through causal inference, capturing the joint interactions between domain and class. Recent UDA methods [ 8 , 9 , 10 , 11 , 12 ] primarily identify domain-invariant and domain-specific features by analyzing the marginal distribution P(X) , frequently overlooking the specific requirements of downstream tasks. As shown in Fig. 1a , these traditional UDA methods assume that domain-specific features must remain class-invariant and that class-specific features are synonymous with domain-invariant features. However, this assumption is overly restrictive and lacks generality. In contrast, as illustrated in Fig. 1b , a more general framework for feature disentanglement accounts for features that exhibit both domain-specific and class-specific. In this paper, we revisit UDA and negative transfer through the lens of causal inference based on disentangled representation learning [ 13 , 10 ] . As illustrated in Fig. 1a , class-invariant features that are irrelevant (useless) to the current discriminative task are excluded from our analysis and discussion. The discriminative feature of each sample is generated through the interplay of causal semantic features and non-causal environmental features . We consider the downstream task and argue that non-causal environmental features, despite being domain-specific, still influence downstream discriminative tasks and are the root cause of negative transfer. Consider a practical example in animal image classification: transferring a model from a specific source domain (e.g., a wild environment) where birds are typically seen in the sky, and monkeys are often found in trees. These environmental features are not causally linked to the animal categories but may still exhibit correlations. When the environment evolves towards a new target domain (e.g., a zoo environment), birds are now typically found in trees, while monkeys are often seen on grass or inside enclosures. If the model prediction relies heavily on non-causal environmental features, as the environment shifts towards a new target domain, it will entrap the model in adopting non-causal and incorrect shortcuts based on environmental features, causing negative transfer. The varying cross-domain discriminative correlation (i.e., conditional distribution P(Y|X) ) between dynamic non-causal environmental features and the ground truth is the primary cause of negative transfer. We term this phenomenon environmental disagreement . To characterize and address it, our causal disentanglement framework incorporates the downstream task by disentangling the discriminative features of each sample x into: 1) semantic features , which are causal factors that maintain domain-invariant correlations with the semantic ground truth, and 2) environmental features , which are non-causally linked and exhibit domain-specific correlations with the semantic ground truth. Both of them correlate with the semantic ground truth Y : semantic features maintain domain-invariant and causal correlations, whereas environmental features exhibit domain-specific and non-causal correlations. Building on this analysis, we propose a new upper bound for the target expected error, extending the work of [ 14 ] and [ 15 ] , to explicitly account for negative transfer using causal inference. Specifically, we causally disentangle each sample into causal semantic features and non-causal environmental features. We define a transition matrix M to quantify the discriminative disagreement between cross-domain environmental features, which we term environmental disagreement . The target expected error is primarily upper-bounded by the source domain empirical error, cross-domain distribution discrepancy, and environmental disagreement. Previous UDA approaches primarily concentrate on the first two terms, often neglecting environmental disagreement, which indicates the extent of negative transfer. Therefore, these theoretical insights motivate an intuitive solution to mitigate negative transfer: learning a model with less non-causal environmental disagreement. According to our analysis and theoretical insights, we propose an innovative UDA approach called Reducing Environmental Disagreement (RED) to mitigate negative transfer. Specifically, we utilize a shared domain-invariant feature extractor to extract domain-invariant causal semantic features, alongside two domain-specific extractors for extracting domain-specific non-causal environmental features from the source and target domains, respectively. To enforce disentanglement, these environmental feature extractors are adversarially trained in the reversed domain: the source environmental feature extractor facilitates classification in the source domain while hindering it in the target domain, and vice versa for the target environmental feature extractor. The transition matrix M is estimated by measuring the correlation of predictions from a shared classifier across domain-specific environmental features. Its trace reflects the conflict within the shared classifier between domain-specific environmental features, thereby quantifying the environmental disagreement. By explicitly reducing this environmental disagreement, the negative transfer can be mitigated. To evaluate RED, we conduct experiments on three widely used and challenging visual benchmarks: Office-31 [ 16 ] , OfficeHome [ 17 ] , and DomainNet [ 18 ] . We integrate the RED framework with various visual backbones, including ResNet [ 19 ] and transformer-based architectures such as Data-efficient Image Transformer (DeiT) [ 20 ] and Vision Transformer (ViT) [ 21 ] . Our method outperforms all compared state-of-the-art (SOTA) baselines under fair experimental setting. Furthermore, when