Closing the Confusion Loop: CLIP-Guided Alignment for Source-Free Domain Adaptation Shanshan Wang, Ziying Feng, Xiaozheng Shen, Xun Yang*, Pichao Wang, Zhenwei He, Xingyi Zhang Abstract Source-Free Domain Adaptation (SFDA) tackles the problem of adapting a pre-trained source model to an unlabeled target domain without accessing any source data, which is quite suitable for the field of data security. Although recent advances have shown that pseudo-labeling strategies can be effective, they often fail in fine-grained scenarios due to subtle inter-class similarities. A critical but underexplored issue is the presence of asymmetric and dynamic class confusion —where visually similar classes are unequally and inconsistently misclassified by the source model. Existing methods typically ignore such confusion patterns, leading to noisy pseudo-labels and poor target discrimination. To address this, we propose CLIP-Guided Alignment (CGA) , a novel framework that explicitly models and mitigates class confusion in SFDA. Generally, our method consists of three parts: (1) MCA: detects first directional confusion pairs by analyzing the predictions of the source model in the target domain; (2) MCC: leverages CLIP to construct confusion-aware textual prompts ( e.g. , “a truck that looks like a bus”), enabling more context-sensitive pseudo-labeling; and (3) FAM: builds confusion-guided feature banks for both CLIP and the source model and aligns them using contrastive learning to reduce ambiguity in the representation space. Extensive experiments on various datasets demonstrate that CGA consistently outperforms state-of-the-art SFDA methods, with especially notable gains in confusion-prone and fine-grained scenarios. Our results highlight the importance of explicitly modeling inter-class confusion for effective source-free adaptation. Our code can be find at https://github.com/soloiro/CGA I Introduction Unsupervised domain Adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain, enabling models to generalize across domain shifts without requiring manual annotation in the target domain [ 46 ] [ 11 ] [ 17 ] . Conventional UDA methods typically assume full access to source data and jointly optimize source and target representations [ 44 ] [ 47 ] [ 5 ] . However, in many real-world scenarios, such as those involving privacy, security, or intellectual property, direct access to source data may be legally or practically infeasible [ 27 ] . e.g. , in applications involving medical imaging, personal biometrics, or financial records, sharing raw data from the source domain - even for model adaptation purposes - could violate confidentiality or privacy. Such cases have led to the emergence of Source-Free Domain Adaptation (SFDA) [ 23 ] [ 28 ] [ 19 ] [ 34 ] , a more realistic and secure setting in which only a pre-trained source model is provided for target adaptation, without access to the original source data. This approach significantly reduces the attack surface for privacy breaches while maintaining the practical benefits of domain adaptation [ 37 ] [ 2 ] [ 55 ] . Fig. 1: During conventional mutual learning, cross-domain misconceptions from the source model contaminate the generalized knowledge in CLIP. Our method which employ confusing-class prompts to induce biased knowledge in CLIP can prevent contamination from cross-domain misconceptions. Despite promising progress, SFDA remains highly challenging due to its ill-posed nature. Without source supervision, the adaptation process is largely based on the predictions of the source model in the target domain. Most existing SFDA methods adopt a pseudo-labeling paradigm, where soft predictions from the source model are refined through clustering or self-training [ 14 ] . These approaches typically rely on the cluster assumption, that is, the belief that the target features form well-separated clusters aligned with semantic classes. However, this assumption often fails in fine-grained settings, where categories exhibit only subtle visual differences. In such cases, noisy pseudo-labels arising from inter-class ambiguity can severely degrade model adaptation [ 3 ] . A core yet underexplored reason for this degradation lies in the asymmetric and dynamic nature of class confusion in the target domain [ 36 ] . Specifically, certain classes are inherently more prone to confusion due to visual similarity ( e.g. , truck V.S. bus), and on specific datasets the confusion is often one-directional ( e.g. , a truck might be misclassified as a bus , but rarely vice versa due to disparities in training data volume, ). Existing SFDA methods largely overlook this asymmetry and typically treat all inter-class relations equally, thus ignoring the nuanced, instance-specific confusion structure that evolves during adaptation [ 56 ] . This mismatch not only leads to incorrect pseudo-labels, but also distorts the learned representation space. Furthermore, confusion between classes in the target domain could even pose a potential privacy risk [ 26 ] [ 10 ] , increasing the urgency to address this issue. To alleviate this problem, we propose a principled framework called CLIP-Guided Alignment (CGA) to explicitly detect, model and resolve the class confusion of the source model in SFDA setting. Instead of passively tolerating noisy pseudo-labels, CGA places class confusion at the center of the adaptation process and addresses it through three tightly coupled stages: confusion perception , confusion representation , and confusion alignment . (1) Confusion perception: identification of directional confusion patterns. We begin by analyzing the soft predictions of the source model over the entire target dataset, constructing a directed and asymmetric confusion graph that reveals which pairs of classes are frequently confused. For example, we may find that many “truck” samples are misclassified as “bus”, forming a high-weighted edge in the confusion graph. Unlike prior works that rely on fixed or symmetric relations, our approach dynamically captures these confusion patterns in a data-driven and evolving fashion, adapting to the model’s behavior throughout training. (2) Confusion representation: encoding ambiguity with CLIP-guided prompts. To improve the quality of supervision under class ambiguity, we leverage the generalization power of Contrastive Language–Image Pre-training (CLIP) [ 31 ] . For each pair of confusion, we construct hybrid textual prompts such as “a truck that looks like a bus ” to represent fine-grained visual ambiguity. These prompts form a set of confusion-aware textual prototypes in CLIP’s semantic space. By projecting target images onto these prototypes, we can obtain refined, context-aware pseudo-labels that not only explicitly model class ambiguity—far more reliable than standard CLIP zero-shot predictions or raw source model outputs but also effectively prevent contamination of the CLIP model by pseudo-labels containing cross-domain misconceptions exported from the source model, as shown in Fig. 1 . (3) Confusion alignment: transferring semantics via contrastive feature matching. To resolve confusion at the representation level, we first construct two confusion-aware feature banks, one from the source model and one from CLIP, by aggregating feature centroids over each confusion-aware pseudo-class. We then perform contrastive alignment between these banks to implicitly transfer CLIP’s semantic priors to the feature space of the source model. This alignment step reduces inter-class overlap in the target domain, producing more discriminative and semantically structured representations without requiring any source data. Together, these three stages form a closed loop pipeline: class confusion is first perceived , then encoded into refined supervision signals, and finally resolved via feature-space alignment. This design enables CGA to systematically mitigate the dominant source of adaptation errors in SFDA between visually similar classes. Our main contributions are summarized as follows. • We uncover the problem of dynamic, asymmetric inter-class confusion in SFDA, and demonstrate its detrimental effect on pseudo-labeling and feature learning in fine-grained settings. • We propose CGA , a novel framework that addresses class confusion through three synergistic stages: confusion detection, CLIP-guided prompt encoding and alignment of features sensitive to confusion, without requiring access to source data. • We design a multi-prototype prompting strategy that could transform CLIP into a context-sensitive supervision assistant, allowing pseudo-labels to reflect nuanced class ambiguity via textual hybridization ( , e.g. , “a truck that looks like a bus”) by adding confusion class text. • We introduce a confusion-aware contrastive alignment mechanism that bridges the feature space of the source model with CLIP’s semantic priors, leading to more structured and discriminative target representations. • CGA establishes new state-of-the-art performance on four widely used SFDA benchmarks, with significant improvements in confusion-prone scenarios, validating the necessity and effectiveness of explicitly modeling class confusion. II RELATED WORK II-A Source-Free Domain Adaptation. Recently, the primary direction in SFDA methods has been the use of pseudo-labels for model self-training [ 49 ] [ 18 ] [ 8 ] . They were based on the cluster assumption, which assumed that the source model has some generalizability in the target domain, which means that samples with the same label should be close in the feature space [ 52 ] . Therefore, fine-tuning can be conducted by exploring its features or outputs on the target data. For example, CoWA-JMDS [ 21 ] obtained pseudo-labels using Gaussian mixture models (GMM) and further mined the knowledge in the target domain using weighted mixture. PLUE [ 25 ] estimated the confidence of the samples by calculating the probability entropy of neighboring samples in the feature space. CRS [ 56 ] considered calculating category similarity using the classifier weights of the source model. However, the similarity relationships it derived are bidirectional and fixed and did not account for the bidirectional and dynamically evolving nature of class confusion relationships during training. Instead, we introduce dynamic confusion estimation into self-training to biased leverage external knowledge from CLIP to improve pseudo-labels. II-B Multimodal Pre-trained Neural Network. Multimodal pre-trained models, with CLIP [ 31 ] as a prominent example, have been extensively utilized in DA due to their remarkable zero-shot capabilities and great generalization power [ 51 ] . Typically, these models were customized through textual and visual prompts [ 57 ] [ 16 ] or adapter [ 12 ] , enabling them to achieve enhanced performance by modifying a few learnable parameters. DAPL [ 13 ] , fine-tuned CLIP using prompt techniques to effectively leverage domain-specific features, aided in domain generalization. Other UDA methods, such as PDA [ 1 ] , have proposed a prompt-based distribution alignment approach that reduced the distribution gap between the source and target domains using a multimodal strategy, thus improving the effectiveness of UDA. In the context of SFDA, the application of multimodal pre-trained models for task optimization is still relatively limited. DIFO [ 41 ] has pioneered the exploration of the potential of the ViL model in SFDA, customized the CLIP model and employed the method to extract specific effective knowledge from multimodal data, thus enhancing the performance of SFDA. ProDe [ 40 ] investigated the noise impact of CLIP on the source model and proposed a proxy denoising approach that effectively leverages CLIP’s knowledge. Furthermore, although feature alignment was a common and effective approach in DA [ 48 ] [ 45 ] [ 6 ] , few methods can effectively optimize the feature space of the source model using CLIP, due to the significant gap between the CLIP and the source model in the feature space. Oriented by classification probabilities, we construct two confusion-aware class center feature banks and implicitly align these banks to refine the feature space of the source model with CLIP’s knowledge. Fig. 2: (a). The modules highlighted in red represent the three core components of our model: MCA (Model Class Confusion Analysis Module), MCC (Multi-Prototype Confused CLIP), and FAM (Feature Space Alignment Module). (b). The detailed workflow of CCF (Constructing a Confusion-Aware Feature Center Bank). (c). The specific structure and data flow of FAM (Feature space Align Module). III Method Let \mathcal{D}_{s} = \{{\mathcal{X}_{s},\mathcal{Y}_{s}}\} = \{({\mathit{x}^{s}_{i},\mathit{y}^{s}_{i}})\}^{N_{s}}_{i=1} be the unseen source domain dateset, where N_{s} is the number of source samples, and we define the target domain dataset as \mathcal{D}_{t} = \{\mathcal{X}_{t}\} = \{\mathit{x}^{t}_{j}\}^{N_{t}}_{j=1} , where N_{t} is the size of the target dateset. In SFDA, the source data cannot be accessed; we could just access the source model f_{\theta_{s}} : x_{s}\rightarrow y_{s} , which has undergone extensive pre-training on \mathcal{D}_{s} . In general, f_{\theta_{s}} is composed of a feature extractor F and a classifier G that is usually based on the fully connected layer. We use the CLIP which consists of an image encoder \mathcal{I} and a text encoder \mathcal{T} . During the prediction phase, the CLIP model generates predictions for zero-shot learning by calculating the cosine similarity between text and image features. In our method, during the training process, we freeze the vast majority of parameters in the CLIP model and only fine-tune the model like CoOp [ 57 ] through the text prompt. The text used to compute the text feature can be represented in the following form:”[prefix][cls]”. where [cls] is the classname and we use ”A picture of a” as the initial prefix. Inputs : Target dataset \mathcal{D}_{t} , Target data x_{i} , source model f_{\theta_{s}} , Multimodal model Clip ; for epoch =1 to N_{e} do for i\leftarrow 0 to \|\mathcal{D}_{t}\| do P^{s}_{i}=f_{\theta_{s}}(x_{i}) , P^{c}_{i}=Clip(x_{i}) ; w^{s}_{i},w^{c}_{i}\leftarrow\textbf{Equation(1)} ; end for {CM^{s},CM^{c}}\leftarrow\textbf{Equation(2)} ; \mathcal{M}\leftarrow\textbf{Equation(3) and (4)} ; \text{textual prompts}\leftarrow\mathcal{M} ; p^{c}_{t}\leftarrow\textbf{Equation(6)} ; F_{c}\leftarrow\textbf{Equation(9) and (10)} ; Train // Algorithm 2 end for Algorithm 1 Model Class Confusion Process Inputs : Target dataset \mathcal{D}_{t} , Target data x_{i} , source model f_{\theta_{s}} , Multi-center prediction p^{c}_{t} ; for epoch =1 to N_{e} do Pre-operation // Algorithm 1 for i\leftarrow 0 to \|\mathcal{D}_{t}\| do p^{s}_{t}=f_{\theta_{s}}(x_{i}) ; p^{r}_{t}\leftarrow\textbf{Equation(13)} ; Update prompts parameters by minimize \mathcal{L}_{clip} ; Using feature alignment calculating F_{s} and F_{c} ; Calculating F_{a} by Equation(12) ; Update parameters \theta_{s} by minimize \mathcal{L}_{s} ; end for end for Algorithm 2 Training Process The general architecture of our method can be understood from Fig. 2 (a). According to the three tightly coupled stages: confusion perception, confusion representation, and confusion alignment, our CGA model mainly comprises three core components: 1. Model Class Confusion Analysis Module (MCA) : This component quantitatively evaluates the class confusion relationships of the model via a confusion graph, or more specifically, a confusion matrix generated by leveraging the full-sample classification results to of the target dataset across corresponding models; 2. Multi-Prototype Confused CLIP (MCC) : Based on the class confusion relationships calculated, we employ the multi-prototype prompting strategy to generate the corresponding confusing class texts, then upgrade the CLIP classification method by adding the confusing category text features into the base text features; 3. Feature Space Alignment Module (FAM) : Building upon the class confusion relationships, this module adaptively weights the target domain features extracted by the source model using classification confidence scores from both the CLIP and the source model to get the center bank of the confusion class features. Then by aligning the projections of target samples calculated by the class feature center banks of the two models, we can implicitly align the feature spaces between two models, thereby transferring CLIP’s knowledge at the feature level to optimize the source model. III-A Model Class Confusion Analysis Module To estimate potential asymmetrical class confusions in the source model and CLIP, we first propose a method to quantitatively estimate the class confusion relationship matrix. We discovered that the class confusion relationships matrix in the model can be inferred by computing class-conditional probability centroids on the target domain. This is because the probabilistic predictions of target domain data inherently encode inter-class confusion patterns, which reflect overlaps and ambiguities between classes. Inspired by SHOT [ 24 ] , we computed probabilistic centers using a method similar to its approach to compute feature centers. However, our method is not entirely identical to SHOT, which computes feature-based centers; instead, we weight probabilities (rather than features) to get the final probabilistic centers. In particular, we only assign weights to the most probable candidate classes for each sample, ignoring the classes to which it cannot possibly belong. Consequently,as shown in Eq. 1 we calculate the contribution weights \boldsymbol{w}_{t} of each sample x_{t} to all classes. {w}_{t}^{k}=\begin{cases}P_{k}(x_{t})&\text{if }P_{k}(x_{t})\in\text{top }N\text{ of }P(x_{t})\\ 0&\text{otherwise},\end{cases} (1) where {P=\delta\circ f_{\theta_{s}}} denotes the probability calculated by the source model f_{\theta_{s}} and the softmax function \delta . P_{k}(x_{t}) denotes the k_{th} component of P(x_{t}) , representing the probability that x_{t} belongs to the k_{th} class. And to eliminate interference, we posit that each probability output contributes exclusively to its N most likely categories. And {w}_{t}^{k} is the result k_{th} of \boldsymbol{w}_{t} . We used the scalar \boldsymbol{w}_{t} to weight each sample, thus deriving the probability center \boldsymbol{{CM}_{k}} , which represents the center of the class for the class k_{th} , and the matrix \boldsymbol{CM} could be interpreted as the confusion class relationship matrix. \boldsymbol{{CM}_{k}}=\frac{\Sigma_{x_{t}\in\mathcal{X}_{t}}{w_{t}^{k}}P(x_{t})}{\Sigma_{x_{t}\in\mathcal{X}_{t}}{w_{t}^{k}}} (2) Considering that if the \boldsymbol{{CM}_{ki}} ( i\neq k )is bigger than \frac{1}{C} ( C is the number of classes), then we can infer that there is a certain degree of confusion between class k and class i . So, from the matrix \boldsymbol{CM} , we generate ordered class confusion pairs based on elements in the matrix that exceed \frac{1}{C} . For example, if \boldsymbol{{CM}_{ki}}>\frac{1}{C} , we create a confusion pair where i is the primary class and j is the secondary class. Subsequently, we compile all the asymmetrical (i\neq j) obtained into a collective set \mathcal{M}_{pre} : \mathcal{M}_{sim}=\{\mid{\boldsymbol{{CM}_{ij}}\geq T_{sim}\kern 5.0ptand\;i\neq j\}} (3) where T_{sim}=max(\frac{1}{C},K_{C}) serves to constrain the size of the set \mathcal{M}_{sim} , with K_{C} denoting the C_{th} largest value among all CM_{ij}(i\neq j) entries. In addition, we add all pairs with i=j to \mathcal{M}_{sim} to facilitate subsequent operations. \mathcal{M}=\mathcal{M}_{sim}\cup\{{}\mid i=1,2,...,C\} (4) Based on the aforementioned computational approach, we can obtain the class confusion relation matrices \boldsymbol{CM^{c}} and \boldsymbol{CM^{s}} for the CLIP and the source model, along with the ordered confusion class pair sets \mathcal{M}^{c} and \mathcal{M}^{s} . III-B Multi-Center CLIP After getting the confusion pairs set \mathcal{M}^{s} , we aim to customize CLIP with it to get the soft predictions that could mitigate the asymmetrical class confusion patterns observed in the source model. Therefore, we aim to achieve our goal by modifying the text input side of CLIP. To match the asymmetrical class confusion, for each element in \mathcal{M}^{s} , we can generate the corresponding class texts. In CLIP, during zero-shot classification of images, the image features F^{I}=\mathcal{I}(x_{t}) are first calculated, then the similarity is calculated with the text features F^{T} generated from the given category descriptions using cosine similarity, generating the classification results. However, in our approach, we use \mathcal{M}^{s} to generate class labels, which also include confused class labels. For instance, for in \mathcal{M}^{s} , we generate the class text ” class(i) ”, and for where i\neq j , we create the confused class text ” class(i) looks like a class(j) ”, where class is the function which can get the corresponding class text. Following this, we categorize the acquired features, clustering text features that share the same principal category into the same text feature subsets. \boldsymbol{\mathcal{F}^{T}_{i}}=\mathcal{T}(\mathcal{G}(\{\mid\in\mathcal{M}^{s}\kern 5.0ptand\kern 5.0ptj=1,2,...,C\})) (5) where \mathcal{G} is the function that translates the similarity relationships within the set \mathcal{M} into text form. We then compute the similarity with the image features for each element in the subset and select the highest similarity score as the logits for that class. The final classification result {\boldsymbol{p^{c}_{t}}} is achieved by subsequently applying the softmax function to these logits. \displaystyle logit_{i}=max({cos(F^{I},\boldsymbol{\mathcal{F}^{T}_{i}})}) (6) \displaystyle\boldsymbol{p^{c}_{t}}=\delta(logit_{1},logit_{2},...,logit_{C}) where cos is the function of the cosine similarity calculation function. And we consider \boldsymbol{p^{c}_{t}} to be a classification result with finer-grained discriminative power compared to the original output of CLIP, so reducing the discrepancy between \boldsymbol{p^{c}_{t}} and the output of the source model could to some extent alleviate the confusion relationship in the source model so that we can extract knowledge from CLIP in a more targeted manner. To summarize the method described above, we define the process of computing \boldsymbol{p^{c}_{t}} from input target data x_{t} as \boldsymbol{p^{c}_{t}}=P^{\prime}(x_{t}) We noticed that the CLIP may yield incorrect classification results for the confusion classes text features. Furthermore, for the degree of similarity between the primary and secondary classes within confused classes, we need to provide quantitative guidance for CLIP. For instance, with the confusion class ”a class[i] looks like a class[j]”, we still require fine-grained adjustments to the specific similarity between the two classes. Therefore, we calculate the similarity of the confusion classes based on the result of Eq. 2 . For each \in\mathcal{M} ( i\neq j ), we could first calculate the degree of similarity between the i_{th} class and the j_{th} through the ratio \mathcal{R}_{sim(i,j)} between {\boldsymbol{CM}}_{ii} and {\boldsymbol{CM}}_{ij} . \mathcal{R}_{sim(i,j)}=\frac{\boldsymbol{CM}_{ii}}{\boldsymbol{CM}_{ij}} (7) Subsequently, we use \mathcal{R}_{sim(i,j)} to estimate the probabilistic centroid \overline{p}_{ij}^{sim} associated with the confusion pair (i,j) that satisfies two conditions: (1) It has a value of zero at all positions except the positions of i_{th} and j_{th} . (2) The sum of the values at the i_{th} and the j_{th} positions is equal to one, with their ratio being \mathcal{R}_{sim(i,j)} . For the non-confusion class \in\mathcal{M} , we construct a probability vector that is one at the i_{th} position and zero at all other positions. Then, we achieve fine-grained adjustment of the confusion class features within CLIP by minimizing the following loss \mathcal{L}_{r} : \mathcal{L}_{r}=KL(\delta(cos(F^{T}_{ij},\boldsymbol{\mathcal{F}^{T}})),\overline{p}_{ij}^{sim}) (8) First, we utilize the text features of the confusion classes F^{T}_{ij}=\mathcal{T}(\mathcal{G}()) in place of the image features, and classify them using the base non-confusion class text features \boldsymbol{\mathcal{F}^{T}}=\mathcal{T}(\mathcal{G}(\{\mid i=1,2,...,C\} to obtain the classification results for the confusion classes. Then, we minimize the Kullback-Leibler(KL) divergence between the softmax output of the classification results and \overline{p}_{ij} , achieving accurate and fine-grained adjustments for each distinct pair of confusion classes. III-C Feature Space Alignment Module At the start of each epoch, we compute the class feature centers for both models throughout the target dataset as shown in Fig. 2 (b). In particular, the classes requiring feature center computation are obtained from the set \mathcal{M} , including the confusion classes. However, due to significant representational discrepancies between CLIP’s image features and the source model’s feature space, directly aligning the respective features proves infeasible. To address this, we leverage the features calculated by source model on target data as the common basis, while utilizing classification outcomes from both models to guide the construction of corresponding class centers, respectively. Unlike the class probability computation in Eq. 2 , here our focus shifts to deriving precise feature centers. So we use a new method to calculate the center of features inspired by BMD [ 30 ] . We need to calculate a score function S_{ij} based on the classification probabilities of each sample to facilitate the classification to find the center of the features of in \mathcal{M} . \displaystyle{R}(\boldsymbol{p}) \displaystyle=\frac{\boldsymbol{p}_{i}}{\boldsymbol{p}_{j}} (9) \displaystyle S_{ij}(\boldsymbol{p}) \displaystyle=\frac{1}{2}(\boldsymbol{p_{i}}+\boldsymbol{p_{j}})-\frac{1}{2}|\frac{(R(\boldsymbol{p})-\mathcal{R}_{sim(i,j)})}{(R(\boldsymbol{p})+\mathcal{R}_{sim(i,j)})}| Remark : The score function S_{ij} comprises two components. The first part calculates the sum of p_{i} and p_{j} . And the second part computes the distance between the ratio of sample probabilities R(p) and the corresponding ratio \mathcal{R}_{sim(i,j)} derived from the confusion matrix {\boldsymbol{CM}} , followed by normalization to ensure a symmetric distribution of the ratios around \mathcal{R}_{sim(i,j)} . To elucidate the specific meanings of these two components, the first term ensures that higher predicted probabilities are more likely to belong to either class i or class j , rather than any other categories; the second term guarantees that the calculated metric yields a larger value when the ratio between the probabilities of class i and class j is closer to the target ratio corresponding to our estimated confusion center. Furthermore, since both components are normalized to the range [0,1], they can be directly summed with equal weighting to form the final composite metric. Additionally, when i=j , the function S(p) degenerates into p_{i} , meaning that for the non-confusing classes, the probability of the corresponding class is directly taken as the result. Then we select the features of the top -M scoring samples for each class and compute the center of the features using a uniform weight. \displaystyle\mathcal{M}_{ij} \displaystyle=\mathop{argmax}\limits_{|\mathcal{M}_{ij}|=M\atop x_{t}\in\mathcal{X}_{t}}S_{ij}(f_{\theta}(x_{t})) (10) \displaystyle F_{c} \displaystyle=\frac{1}{M}\sum\limits_{k\in\mathcal{M}_{ij}}F(x_{k}) After getting the center banks, as shown in Fig. 2 (c), during the training phase, we treat each feature center bank as a set of basis vectors that structurally approximate the model’s feature space on the target dataset. Subsequently, using MLP projection mechanisms, we project the input features F_{t} into the corresponding space by computing a weighted combination of the centers in the feature bank, then the projected features are fused with the original features through residual summation within the residual network framework, producing two distinct outputs F_{clip} and F_{s} . Finally, contrastive learning is used to minimize the distance between F_{c} and F_{s} , achieving implicit alignment between CLIP and the source model. \mathcal{L}_{ct}=Simclr(\boldsymbol{F_{s}},\boldsymbol{F_{clip}}) (11) Simclr [ 7 ] is a self-supervised contrastive loss function. The augmented feature F_{a} is obtained by dynamically weighting the F_{clip} and F_{s} . The weight \mathcal{W} is a simple MLP that is connected only to the input feature F_{t} . F_{a}=\mathcal{W}(F_{t})F_{s}+(1-\mathcal{W}(F_{t}))F_{clip} (12) III-D Loss Function To extract more accurate information from \boldsymbol{p^{c}_{t}} and \boldsymbol{p_{t}^{s}}=\delta(f_{\theta}(x)) , we refine the probabilities of these two distributions into a single consolidated probability \boldsymbol{p^{r}_{t}} and subsequently remove its gradient. \boldsymbol{p^{r}_{t}}=\left\{\begin{aligned} \mathop{argmin}\limits_{H(\boldsymbol{p_{t}})}(\boldsymbol{p^{c}_{t}},\boldsymbol{p_{t}^{s}})\quad&,ifA(\boldsymbol{p^{c}_{t}},\boldsymbol{p_{t}^{s}})\\ \omega\boldsymbol{p^{c}_{t}}+(1-\omega)\boldsymbol{p_{t}^{s}}\quad&,otherwise\end{aligned}\right. (13) where H(x)=-\sum_{i=1}^{n}p(x_{i})\log p(x_{i}) calculates the information entropy of probability x and the condition is fulfilled A(x_{1},x_{2}) if and only if argmax(x_{1})=argmax(x_{2}) is satisfied. If the pseudo-labels of \boldsymbol{p_{t}^{s}} and \boldsymbol{p^{c}_{t}} are consistent, then \boldsymbol{p^{r}_{t}} is the one with the lower information entropy between them; otherwise, we use the information entropies of both to perform weighting. \omega=\frac{H(\boldsymbol{p_{t}^{s}})}{H(\boldsymbol{p_{t}^{s}})+H(\boldsymbol{p^{c}_{t}})} (14) After getting \boldsymbol{p^{r}_{t}} in Eq. 13 , we then adopt \boldsymbol{p^{r}_{t}} as the reference distribution to optimize the source model, the FAM, and CLIP’s prompts by minimizing the KL divergence between their respective outputs: \displaystyle\mathcal{L}_{c}= \displaystyle KL(\boldsymbol{{p}^{c}_{t}},\boldsymbol{{p}^{r}_{t}}) (15) \displaystyle\mathcal{L}_{s}= \displaystyle KL(\boldsymbol{{p}^{s}_{t}},\boldsymbol{{p}^{r}_{t}}) \displaystyle\mathcal{L}_{a}= \displaystyle KL(\boldsymbol{\delta(G(F_{a}))},\boldsymbol{{p}^{r}_{t}}) The loss function of CLIP \mathcal{L}_{clip} consists of the downstream task alignment loss \mathcal{L}_{c} and the text refinement loss \mathcal{L}_{r} shown below: \mathcal{L}_{clip}=\mathcal{L}_{r}+\mathcal{L}_{c} (16) The loss \mathcal{L}_{source} of the source model and FAM consists of \mathcal{L}_{s} to align \boldsymbol{{p}^{s}_{t}} and \boldsymbol{{p}^{r}_{t}} , optimizing the source model using CLIP’s knowledge, \mathcal{L}_{a} for optimizing the FAM and \mathcal{L}_{ct} to implicitly align the feature spaces of the source model and CLIP. \mathcal{L}_{source}=\mathcal{L}_{s}+\alpha\mathcal{L}_{a}+\gamma\mathcal{L}_{ct} (17) where \alpha , \beta and \gamma are the hyper-parameters. In general, our method consists of two processes: a pre-processing process at the beginning of each epoch and the training process, which are detailed in Algorithm 1 and Algorithm 2 , respectively. All operations in Algorithm 1 are recomputed at the beginning of each epoch to dynamically adapt to the evolving source model during training. Upon completion of training, we exclusively employ the source model (typically ResNet) for downstream tasks or testing, while discarding both CLIP and the FAM network as they solely serve auxiliary roles during the training phase. TABLE I: Accuracy (%) on VisDA for UDA and source-free UDA methods (ResNet-101). Method SF CLIP plane bcycl bus car horse knife mcycl person plant sktbrd train truck Avg. DAPL-RN [ 13 ] 97.8 83.1 88.8 77.9 97.4 91.5 94.2 79.7 88.6 89.3 92.5 62.0 86.9 PADCLIP-RN [ 20 ] 96.7 88.8 87.0 82.8 97.1 93.0 91.3 83.0 95.5 91.8 91.5 63.0 88.5 ADCLIP-RN [ 35 ] 98.1 83.6 91.2 76.6 98.1 93.4 96.0 81.4 86.4 91.5 92.1 64.2 87.7 PDA-R [ 1 ] 97.2 82.3 89.4 76.0 97.4 87.5 95.8 79.6 87.2 89.0 93.3 62.1 86.4 DAMP-R [ 9 ] 97.3 91.6 89.1 76.4 97.5 94.0 92.3 84.5 91.2 88.1 91.2 67.0 88.4 SHOT [ 24 ] 95.0 87.4 80.9 57.6 93.9 94.1 79.4 80.4 90.9 89.8 85.8 57.5 82.7 NRC [ 50 ] 96.8 91.3 82.4 62.4 96.2 95.9 86.1 90.7 94.8 94.1 90.4 59.7 85.9 GKD [ 39 ] 95.3 87.6 81.7 58.1 93.9 94.0 80.0 80.0 91.2 91.0 86.9 56.1 83.0 AaD [ 49 ] 97.4 90.5 80.8 76.2 97.3 96.1 89.8 82.9 95.5 93.0 92.0 64.7 88.0 AdaCon [ 4 ] 97.0 84.7 84.0 77.3 96.7 93.8 91.9 84.8 94.3 93.1 94.1 49.7 86.8 CoWA [ 21 ] 96.2 89.7 83.9 73.8 96.4 97.4 89.3 86.8 94.6 92.1 88.7 53.8 86.9 ELR [ 53 ] 97.1 89.7 82.7 62.0 96.2 97.0 87.6 81.2 93.7 94.1 90.2 58.6 85.8 PLUE [ 25 ] 94.4 91.7 89.0 70.5 96.6 94.9 92.2 88.8 92.9 95.3 91.4 61.6 88.3 CPD [ 58 ] 96.7 88.5 79.6 69.0 95.9 96.3 87.3 83.3 94.4 92.9 87.0 58.7 85.5 TPDS [ 38 ] 97.6 91.5 89.7 83.4 97.5 96.3 92.2 82.4 96.0 94.1 90.9 40.4 87.6 DIFO-V [ 41 ] 97.5 89.0 90.8 83.5 97.8 97.3 93.2 83.5 95.2 96.8 93.7 65.9 90.3 ProDe-V [ 40 ] 98.3 92.0 87.3 84.4 98.5 97.5 94.0 86.4 95.0 96.1 94.2 75.6 91.6 CGA(ours) 98.9 91.6 87.5 88.2 99.2 98.3 94.9 87.5 96.0 95.0 94.4 74.5 92.2 TABLE II: Accuracy (%) on Office-Home for UDA and source-free UDA methods (ResNet-50). Method SF CLIP A→C A→P A→R C→A C→P C→R P→A P→C P→R R→A R→C R→P Avg. DAPL-RN [ 13 ] 54.1 84.3 84.8 74.4 83.7 85.0 74.5 54.6 84.8 75.2 54.7 83.8 74.5 PADCLIP-RN [ 20 ] 57.5 84.0 83.8 77.8 85.5 84.7 76.3 59.2 85.4 78.1 60.2 86.7 76.6 ADCLIP-RN [ 35 ] 55.4 85.2 85.6 76.1 85.8 86.2 76.7 56.1 85.4 76.8 56.1 85.5 75.9 PDA-R [ 1 ] 55.4 85.1 85.8 75.2 85.2 85.2 74.2 55.2 85.8 74.7 55.8 86.3 75.3 DAMP-R [ 9 ] 59.7 88.5 86.8 76.6 88.9 87.0 76.3 59.6 87.1 77.0 61.0 89.9 78.2 SHOT [ 24 ] 56.7 77.9 80.6 68.0 78.0 79.4 67.9 54.5 82.3 74.2 58.6 84.5 71.9 NRC [ 50 ] 57.7 80.3 82.0 68.1 79.8 78.6 65.3 56.4 83.0 71.0 58.6 85.6 72.2 GKD [ 39 ] 56.5 78.2 81.8 68.7 78.9 79.1 67.6 54.8 82.6 74.4 58.5 84.8 72.2 AaD [ 49 ] 59.3 79.3 82.1 68.9 79.8 79.5 67.2 57.4 83.1 72.1 58.5 85.4 72.7 AdaCon [ 4 ] 47.2 75.1 75.5 60.7 73.3 73.2 60.2 45.2 76.6 65.6 48.3 79.1 65.0 CoWA [ 21 ] 56.9 78.4 81.0 69.1 80.0 79.9 67.7 57.2 82.4 72.8 60.5 84.5 72.5 ELR [ 53 ] 58.4 78.7 81.5 69.2 79.5 79.3 66.3 58.0 82.6 73.4 59.8 85.1 72.6 PLUE [ 25 ] 49.1 73.5 78.2 62.9 73.5 74.5 62.2 48.3 78.6 68.6 51.8 81.5 66.9 CPD [ 58 ] 59.1 79.0 82.4 68.5 79.7 79.5 67.9 57.9 82.8 73.8 61.2 84.6 73.0 TPDS [ 38 ] 59.3 80.3 82.1 70.6 79.4 80.9 69.8 56.8 82.1 74.5 61.2 85.3 73.5 DIFO-V [ 41 ] 70.6 90.6 88.8 82.5 90.6 88.8 80.9 70.1 88.9 83.4 70.5 91.2 83.1 ProDe-V [ 40 ] 74.6 92.9 92.4 84.4 93.0 92.2 83.8 74.8 92.4 84.9 75.2 93.7 86.2 CGA(ours) 74.1 92.8 92.6 86.0 93.1 92.4 86.0 73.3 92.2 86.0 73.0 93.6 86.3 TABLE III: Accuracy (%) on DomainNet-126 for UDA and source-free UDA methods (ResNet-50). Method SF CLIP C→P C→R C→S P→C P→R P→S R→C R→P R→S S→C S→P S→R Avg. DAPL-RN [ 13 ] 72.4 87.6 65.9 72.7 87.6 65.6 73.2 72.4 66.2 73.8 72.9 87.8 74.8 ADCLIP-RN [ 35 ] 71.7 88.1 66.0 73.2 86.9 65.2 73.6 73.0 68.4 72.3 74.2 89.3 75.2 DAMP-R [ 9 ] 76.7 88.5 71.7 74.2 88.7 70.8 74.4 75.7 70.5 74.9 76.1 88.2 77.5 SHOT [ 24 ] 63.5 78.2 59.5 67.9 81.3 61.7 67.7 67.6 57.8 70.2 64.0 78.0 68.1 NRC [ 50 ] 62.6 77.1 58.3 62.9 81.3 60.7 64.7 69.4 58.7 69.4 65.8 78.7 67.5 GKD [ 39 ] 61.4 77.4 60.3 69.6 81.4 63.2 68.3 68.4 59.5 71.5 65.2 77.6 68.7 AdaCon [ 4 ] 60.8 74.8 55.9 62.2 78.3 58.2 63.1 68.1 55.6 67.1 66.0 75.4 65.4 CoWA [ 21 ] 64.6 80.6 60.6 66.2 79.8 60.8 69.0 67.2 60.0 69.0 65.8 79.9 68.6 PLUE [ 25 ] 59.8 74.0 56.0 61.6 78.5 57.9 61.6 65.9 53.8 67.5 64.3 76.0 64.7 TPDS [ 38 ] 62.9 77.1 59.8 65.6 79.0 61.5 66.4 67.0 58.2 68.6 64.3 75.3 67.1 DIFO-V [ 41 ] 76.6 87.2 74.9 80.0 87.4 75.6 80.8 77.3 75.5 80.5 76.7 87.3 80.0 ProDe-V [ 40 ] 83.2 92.4 79.0 85.0 92.3 79.3 85.5 83.1 79.1 85.5 83.4 92.4 85.0 CGA(ours) 85.5 90.5 83.2 85.1 91.1 84.2 87.0 86.0 83.7 86.2 84.9 91.5 86.6 TABLE IV: Accuracy (%) on Office-31 for UDA and source-free UDA methods (ResNet-50). Method SF C A \rightarrow D A \rightarrow W D \rightarrow A D \rightarrow W W \rightarrow A W \rightarrow D AVG. Source 79.1 76.9 59.9 95.5 61.4 98.8 78.6 SHOT [ 24 ] 93.7 91.1 74.2 98.2 74.6 100. 88.6 NRC [ 50 ] 96.0 90.8 75.3 99.0 75.0 100. 89.4 AaD [ 49 ] 96.4 92.1 75.0 99.1 76.5 100. 89.9 AdaCon [ 4 ] 87.7 83.1 73.7 91.3 77.6 72.8 81.0 CoWA [ 21 ] 94.4 95.2 76.2 98.5 77.6 99.8 90.3 ELR [ 53 ] 93.8 93.3 76.2 98.0 76.9 100. 89.6 PLUE [ 25 ] 89.2 88.4 72.8 97.1 69.6 97.9 85.8 TPDS [ 38 ] 97.1 94.5 75.7 98.7 75.5 99.8 90.2 DIFO-V [ 41 ] 97.2 95.5 83.0 97.2 83.2 98.8 92.5 ProDe-V [ 40 ] 96.6 96.4 83.1 96.9 82.9 99.8 92.6 CGA(ours) 97.4 95.7 82.5 98.5 82.0 100. 92.7 IV Experiment IV-A Implementation details To evaluate the effectiveness of the proposed method, we employ experiments on some challenging datasets, including the small-scaled datasets of Office-31 [ 22 ] and Office-Home [ 43 ] , and large-scaled datasets of VisDA [ 29 ] and DmainNet-126 [ 32 ] . We employ ResNet [ 15 ] as the foundational architecture for our model, which consists mainly of a feature extractor and a classification layer. To ensure a fair comparison, we utilize the same ResNet variant as our counterpart for the backbone of our model. Specifically, for Office-31, Office-Home and DomainNet-126, we have chosen ResNet-50 to serve as the backbone. In VisDA, we used ResNet-101. For CLIP, we uniformly use ViT-B/16 as the backbone of the image encoder. We used identical hyperparameter settings in three datasets: parameter \alpha was set to 0.5 for good results, while \gamma was fixed at 0.05 and its sensitivity analysis is presented in Fig. 6 (b). For N in the computation of the confusion matrix in Eq. 1 , we set it to 2 in all settings. And the analysis compared to other settings can be seen in Fig. 6 (a). IV-B Comparison Performance We conducted tests of our method and several comparative approaches in three datasets, and the results are documented in Tables I-IV. In each column, the bold figures indicate the highest accuracy achieved in the respective tasks, and SF and CLIP (or C) indicate whether the method is a source-free setting and whether the CLIP model has been used, respectively. Tables I-IV demonstrate that our method outperforms the previously best methods on average in three datasets. Specifically, of the 12 tasks on Office-Home, 6 achieved optimal results, and of the 12 categories on VisDA, 8 obtained the best performance, with half the configurations on Office-31 being optimal, and in DomainNet-126, 9 settings of 12 settings achieved the best. In particular, on the VisDA dataset, among a group of semantically similar classes: car, bus, train, and truck, we observed an improvement of 3.9% over the previous best method. These results indicate that our CGA method improves cross-domain performance and showcases its fine-grained classification capability for confusable classes. Additionally, we note that while our method achieves modest average improvements on small-scale datasets (Office-31 and Office-Home), it demonstrates significantly more pronounced gains on larger-scale datasets (VisDA and DomainNet-126). This indicates that our approach is particularly well suited for handling large-scale data scenarios. TABLE V: Ablation accuracy (%) on Office-31,Office-Home and VisDA. \mathcal{L}_{c} \mathcal{L}_{ct} \mathcal{L}_{r} Office-31 Office-Home VisDA AVG 92.7 86.2 92.2 90.3 90.9 82.2 90.5 87.9 91.0 83.0 91.3 88.4 91.9 83.6 91.8 89.1 CGA w/o MCC 87.5 84.8 91.4 87.9 V Discussion V-A Ablation study. In Table V , we report the results of the ablation study for various settings in 3 different datasets. We conducted a quantitative analysis of the functional roles of individual modules by removing specific loss components and related methods in the experiment. When we remove \mathcal{L}_{c} , we observe a decrease in the overall performance of the model. Similarly, when we remove \mathcal{L}_{r} , we observe that it results in only a minor performance drop compared to removing \mathcal{L}_{c} . We hypothesize that this occurs because \mathcal{L}_{c} enables CLIP to partially acquire relevant knowledge from the source model, thus mitigating the negative impact caused by the lack of Lr of losing qualitative and quantitative constraints on confusing text classes. When \mathcal{L}_{ct} is removed, the model’s mean Average Precision drops by 2 \% , demonstrating the effectiveness of the feature-space alignment module. Finally, we evaluated the impact of removing MCC (Multi-Prototype CLIP), directly using standard CLIP classification results in our experiments. We found that its impact on the entire experiment was substantial and only achieved a marginal improvement compared to the results of the direct zero-shot classification using CLIP. This demonstrates the effectiveness of MCC in addressing issues caused by class confusion. Fig. 3: The Grad-CAM visualizations of Source, COWA, CGA w/o MCC and CGA models trained in Visda. TABLE VI: Accuracy(%) of CLIP and its fine-tuned variant models in our method. Method Office-31 Office-Home VisDA →A →W →D Avg. →A →C →P →R Avg. Sy→Re CLIP 78.1 79.6 78.9 78.9 81.9 62.9 90.1 88.5 80.8 86.9 MCC 75.3 79.6 76.3 77.1 77.7 56.6 82.5 81.0 75.4 74.7 MCC w \mathcal{L}_{r} 78.9 81.3 83.1 81.1 81.6 65.5 87.8 88.5 80.9 87.7 MCC in CGA 82.1 95.5 94.2 90.6 84.1 69.2 91.2 90.5 83.8 92.2 Fig. 4: Feature distribution visualization comparison on transfer task Ar→Cl in Office-Home by t-SNE. Fig. 5: (a) Time Comparison Between Introduced Pre-operations and Training Process (b) Time Consumption of Feature Alignment Module During Training. Fig. 6: (a). The center calculate method; (b) the sensitivity of \gamma Fig. 7: Prompt GPU Memory Consumption Analysis. In Table VI , we compare the improvements brought by our full method and partial methods to CLIP in our experiments. In the first row of the table, we present the classification accuracy of the original CLIP model across different downstream tasks. Then in the second and third rows, we observed that directly using the Multi-prototype confusion CLIP (MCC) for classification without additional optimization methods actually performed worse. However, simply refining the confusion text with L_{r} led to improved classification accuracy over the original CLIP baseline. In the last row , when we applied the complete method, MCC achieved significantly improved results. This demonstrates that our approach not only achieves significant improvements in optimizing the source model, but also effectively enhances the auxiliary CLIP model through mutual learning. V-B Feature distribution visualization. In the task of A→C within Office-Home, we use the t-SNE tool [ 42 ] to visualize the feature distribution. And we choose 4 settings, including Source (only pre-train source model), CLIP (CLIP’S zero-shot), COWA, and CGA (ours). As shown in Fig. 4 , from left to right, we observe a progressive reduction in inter-class feature overlap, increasingly compact clustering of intra-class features, achieving the best performance in CGA. This substantiates the efficacy of our method in characterizing feature distributions. Fig. 8: The confusion matrix for classification on VisDA. V-C Grad-CAM visualizations. We used Grad-CAM [ 33 ] visualizations to evaluate the fine-grained discriminative capability of our method trained on Visda across similar categories (bus and truck). As shown in Fig. 3 , the source represents the results obtained using a model pre-trained solely on the source domain, while the COWA, CGA w/o MCC and CGA (ours) show the results from different methods. In the BUS visualization, we can observe that the source method focuses more on the front section of the bus, while COWA pays attention to the overall structure of the bus, and the attention map of CGA w/o MCC indicates that its corresponding model exhibits a broader focus area. In contrast, our CGA method emphasizes the more discriminative doors of the bus. Similarly, in the visualization of truck, both Source and COWA mainly concentrate on the overall structure, particularly the truck cab section,and CGA w/o MCC focus a larger areas, whereas our method targets the distinctive carriage components and undercarriage chassis parts of the truck. Therefore, our experiments demonstrate that CGA exhibits a strong fine-grained discriminative capacity among similar classes, with a heightened focus on entity-specific distinctive features. V-D Confusion Matrix. To quantitatively observe the degree of confusions between classes and to demonstrate the asymmetric classification behavior of our model discussed earlier, we computed the true confusion matrix using ground-truth labels (the first row) and the estimated confusion matrix (the second row) via Eq. 2 using pseudo-labels. We conducted experiments on the VisDA using three models: Source, COWA, CGA No dynamic(without dynamically updating confusion relationships and confused class text at each epoch start) and CGA. As shown in Fig. 8 , in the first row ,the true confusion matrix, we can observe that while COWA alleviates some of the class confusion present in the pre-trained model to a certain extent, there are still significant issues between certain classes. In contrast, CGA significantly reduces the class confusion issue, markedly enhancing the fine-grained discriminative capability of the model. Compared with CGA, we can find that CGA No dynamic failed to resolve the confusion between class 3 (car) and class 11 (truck), likely because the initially computed class confusion relationships only captured characteristics from the early training stage. As training progressed and model parameters updated, these fixed relationships became misaligned with the actual model behavior. Meanwhile, it can be observed that the confusion matrix estimated via Eq. 2 exhibits a notable similarity to the true confusion matrix, particularly in the off-diagonal elements. Furthermore, the weights of the off-diagonal elements in our estimated confusion matrix progressively decrease from left to right, further validating the accuracy of our estimated class confusion relationships and the effectiveness of the proposed method in reducing confusion issues. Obviously, nearly all confusion matrices, whether true or estimated, are not strict symmetry, which strongly supports our thesis on the model’s asymmetric prediction behavior. V-E Parameter Sensitivity. Here, we separately discuss the method for calculating the center in Eq. 1 and Eq. 2 , and the influence of the parameter \gamma on the model. As shown in Fig. 6 (a), we compare the setting of effective weight quantity N=2 in our CGA with other methods. We can see the SHOT (which treats each bit of probability as an effective weight, i.e. , N=C), whose performance is suboptimal, likely due to its excessive introduction of interference information, leading to significant errors in both qualitative and quantitative estimation of confused classes. And BMD (similar to Eq. 10 , selecting only the top M probabilities corresponding to the highest ranked classes with the same weight) perform poorly, because it dismisses the inherently inter-class confusion patterns in probability. And our method achieves comparable performance to the CPL [ 54 ] (which dynamically determines effective weights for each sample) with lower computational complexity. And in Fig. 6 (b), our experiments demonstrate that 0.1 is an appropriate choice for the hyperparameter \gamma . V-F Training Time and Memory Usage Analysis. In addition to the cross-entropy loss introduced during training, which followed other classification models, we incorporated additional training losses and parameters, resulting in an increased computational overhead in both space and time for our model. Consequently, we conducted a computational efficiency analysis of the additional components, evaluating both time and space complexity. First, we conducted a temporal analysis of the pre-processing operations performed by CLIP and the source model at the beginning of each epoch. As illustrated in Fig. 5 (a), experiments under the A-C configuration of the Office-Home dataset demonstrate that the additional time overhead introduced by these two pre-operations remains entirely acceptable relative to the training phase, imposing no significant computational burden. Moreover, the additional time overhead incurred by adding confusion prompts is negligible. We subsequently analyzed the time overhead introduced by the feature-align module within a single training batch. As evidenced by Fig. 5 (b), the computational latency attributable to the feature alignment module remains marginal, accounting for merely 16% of the total processing time, during both the forward and backward propagation phases. Then, we analyze the GPU memory consumption induced by the text prompts generated by Eq.(5). As shown in Fig. 7 given their participation in training and substantial amount, these prompts incur significant memory overhead. Under the C→P transfer setting on DomainNet-126, the experimental results demonstrate that: The No MCC (using only category prompts) configuration with 126 prompts occupies 15,043 MB of GPU memory. When using 200 prompts, it occupies 17,783 MB, resulting in an additional 18.2% memory overhead. In contrast, our method (using 252 prompts) occupies 18,725 MB, representing a 24.5% increase in memory consumption. Additionally, we estimated the average GPU memory consumption per prompt using the regression equation shown in Figure. 7 b, and the results indicate that each prompt occupies approximately 29 MB of memory on average. VI Conclusion In this paper, we propose a targeted guidance approach for CLIP by quantitatively estimating the class confusion relationships of the source model on downstream datasets. Through directional prompt engineering, we generate probabilistic outputs that mitigate existing issues in the source model. Simultaneously, we construct a confused feature center bank to implicitly align the feature spaces between models. This dual mechanism enhances the model’s fine-grained discriminative capability among similar classes, further closing the confusion loop in SFDA.