When Domain Generalization meets Generalized Category Discovery: An Adaptive Task-Arithmetic Driven Approach

Vaibhav Rathore, Shubhranil B, Saikat Dutta, Sarthak Mehrotra, Zsolt Kira, Biplab Banerjee · 2025

Read When Domain Generalization meets Generalized Category Discovery: An Adaptive Task-Arithmetic Driven Approach by Vaibhav Rathore, Shubhranil B, Saikat Dutta, Sarthak Mehrotra, Zsolt Kira, Biplab Banerjee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14897.txt.

Introduction

Vaibhav Rathore and co-authors tackle a harder form of image grouping in a 2025 paper. The task is to sort images into known and new classes in a new setting, when only labeled images from a different setting are available for training. Shifts between settings often break existing methods, and peeking at the target setting during training is not always possible.

Their system trains in short episodes that mix real and synthetic settings, so it learns features that do not depend on one setting. A weighting scheme then adjusts the base model based on how each fine-tuned version scores. Tests on three data sets beat adapted rival methods. The approach suits cases where the target images cannot be seen until the model is used.

Catalog record

CREATORVaibhav Rathore, Shubhranil B, Saikat Dutta, Sarthak Mehrotra, Zsolt Kira, Biplab Banerjee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14897

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Domain Generalization meets Generalized Category Discovery: An Adaptive Task-Arithmetic Driven Approach Vaibhav Rathore, Shubhranil B, Saikat Dutta, Sarthak Mehrotra, Zsolt Kira, Biplab Banerjee Abstract Generalized Class Discovery (GCD) clusters base and novel classes in a target domain, using supervision from a source domain with only base classes. Current methods often falter with distribution shifts and typically require access to target data during training, which can sometimes be impractical. To address this issue, we introduce the novel paradigm of Domain Generalization in GCD (DG-GCD), where only source data is available for training, while the target domain—with a distinct data distribution—remains unseen until inference. To this end, our solution, DG 2 CD-Net, aims to construct a domain-independent, discriminative embedding space for GCD. The core innovation is an episodic training strategy that enhances cross-domain generalization by adapting a base model on tasks derived from source and synthetic domains generated by a foundation model. Each episode focuses on a cross-domain GCD task, diversifying task setups over episodes and combining open-set domain adaptation with a novel margin loss and representation learning for optimizing the feature space progressively. To capture the effects of fine-tunings on the base model, we extend task arithmetic by adaptively weighting the local task vectors concerning the fine-tuned models based on their GCD performance on a validation distribution. This episodic update mechanism boosts the adaptability of the base model to unseen targets. Experiments across three datasets confirm that DG 2 CD-Net outperforms existing GCD methods customized for DG-GCD. 1 Introduction Recent advances in deep learning Voulodimos et al. 2018 ; Chai et al. 2021 have improved visual inference models; however, challenges persist in limited supervision, domain shifts, and detecting novel classes in inference. Semi-supervised and self-supervised methods Van Engelen and Hoos 2020 ; Jaiswal et al. 2020 ; Zhai et al. 2019 address limited supervision but assume a closed-set environment. Domain adaptation (DA) and domain generalization (DG) Wang and Deng 2018 ; Zhou et al. 2022 aim to create domain-agnostic embeddings for distribution generalization. However, in open-set applications Saito et al. 2018 ; Bele et al. 2024 , they typically classify novel samples in a single outlier class. Figure 1 : We present a novel variant of GCD, Domain Generalization for Generalized Category Discovery , where a model is trained on a source domain (photo) and evaluated on a target domain (cartoon). During inference, the model must cluster seen classes while also identifying and distinguishing novel classes . Applications like autonomous driving and healthcare require finer categorization of novel samples. Novel Class Discovery (NCD) Hsu et al. 2018 ; Han et al. 2019 addresses this by clustering novel classes in an unlabeled target domain using labeled source data with known classes. Generalized Class Discovery (GCD) Vaze et al. 2022 ; Pu et al. 2023 ; Zhang et al. 2023b extends this to both known and novel classes in the target domain, with the support of source domain knowledge. Cross-Domain GCD (CD-GCD) Rongali et al. 2024 further advances this by handling cases where the source and target domains differ in distribution. Nonetheless, all these settings assume simultaneous access to source and target data to carry out training, which is often impractical. For example, an autonomous vehicle Sun 2022 ; Zhang et al. 2019 trained on California’s roads in summer may face unexpected challenges on snowy Moscow streets. Similarly, medical AI models Yoon et al. 2023 ; Behar et al. 2023 trained on high-quality images may need to analyze lower-resolution scans from rural areas to detect new pathologies. In these cases, the target domains are majorly inaccessible until inference. Thus, robust AI systems must adapt to varying environments, organizing both known and novel data categories without prior exposure and ensuring reliable performance in unexpected scenarios. To meet these challenges, we propose DG-GCD (Table 1 ), a novel and practical setting that leverages a labeled source domain to train a feature extractor on known classes. The task is to leverage the trained model to cluster unlabeled samples from a previously unseen, visually distinct target domain containing both known and novel classes, with an undefined known-novel class ratio—suggesting solving the GCD problem within a DG framework with uncharted domain and semantic shifts between training and testing . Existing GCD and CD-GCD methods perform sub-optimally in DG-GCD, when we train them solely on the source domain without considering the target domain as per DG requirements (Table 1 ). Besides, single-source open-set DG methods Bele et al. 2024 , which generalize from a labeled source to a mixed-class target domain, categorize all novel samples as a single outlier, diverging from DG-GCD’s objectives. Figure 2 : Proposed episodic training : A pre-trained global model is updated using task vectors from n_{e} episode-specific fine-tuned models, leveraging a novel dynamic weighting scheme. This scheme adjusts the task vectors based on their GCD generalization performance on a held-out unseen validation distribution. Our proposal: As a remedy, we present DG-GCD-Net ( aka DG 2 CD-Net). Generalizing from a single-source domain is inherently challenging due to limited data diversity, the need to adapt to unseen novel classes in the target domain, and the varying known-novel class splits that complicate learning. Standard approaches often struggle to effectively handle both domain shifts and class-level variations. To address this, we propose a novel progressive episodic training strategy (Fig. 2 ) designed to enhance a pre-trained global feature encoder. In each episode, the CD-GCD task is tackled locally by using a subset of the source domain with a reduced class set. Additionally, a stylistic variant of the source domain, generated via a diffusion model Brooks et al. 2023 , acts as a pseudo-target domain containing all known classes, but is treated unlabeled. These episodic splits are dynamically generated to simulate diverse conditions. 0.25 0.5 0.75 1 0.96 0.63 0.58 0.54 Fraction of sign conflicts Ties Merging Yadav et al. 2024 Fisher Merging Matena and Raffel 2022 Task Arithmetic Ilharco et al. 2023 DG 2 CD-Net Figure 3 : Sign Conflicts —opposing gradient updates leading to greater parameter divergence across episodes in various model merging methods for our episodic training on PACS. Lower values indicate improved training stability and accuracy. For CD-GCD, known classes from the source (labeled) and synthesized (unlabeled) domains are aligned using an open-set domain adaptation (OSDA) objective, while contrastive learning boosts feature discriminativeness. Existing OSDA methods like Saito et al. 2018 often fail with fine-grained novel classes, causing misalignment with known classes. To address this, we propose a novel margin objective to enforce clearer separation between episode-specific known and novel classes, improving novel class discovery. To aggregate task-specific knowledge from multiple fine-tuned models, we apply task arithmetic (TA) Ilharco et al. 2023 , combining task vectors that capture weight shifts between pre-trained and fine-tuned models to improve adaptability. While certain TA variants focus only on top-ranked parameters Yadav et al. 2024 , this approach leads to instability in our context of multiple parallel episodes with varied data distributions. To maintain consistency, we use TA across the entire weight space. To enhance generalization beyond traditional TA, we introduce a novel dynamically weighted TA approach tailed for DG-GCD, which deals with scaling task vectors according to each fine-tuned model’s generalization performance, assessed through clustering on a

Read the complete text