TRUST: Leveraging Text Robustness for Unsupervised Domain Adaptation
Mattia Litrico, Mario Valerio Giuffrida, Sebastiano Battiato, Devis Tuia · 2025
Read TRUST: Leveraging Text Robustness for Unsupervised Domain Adaptation by Mattia Litrico, Mario Valerio Giuffrida, Sebastiano Battiato, Devis Tuia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06452.txt.
Introduction
Mattia Litrico, Mario Valerio Giuffrida, Sebastiano Battiato, and Devis Tuia present TRUST for unsupervised domain adaptation in two thousand twenty-five. Vision models trained in one domain falter in another with no labels. Hard shifts change both scenes and object looks at once. Language holds up better under such complex shifts.
Captions of target images yield pseudo labels for the vision model. A normalized CLIP score gauges doubt in each such label. Loss weights then mute the harm from bad labels. A soft contrastive loss aligns vision and language spaces too. Tests set new best scores on classic and geographic shifts.
Catalog record
| CREATOR | Mattia Litrico, Mario Valerio Giuffrida, Sebastiano Battiato, Devis Tuia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-06452
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TRUST: Leveraging Text Robustness for Unsupervised Domain Adaptation Mattia Litrico, Mario Valerio Giuffrida, Sebastiano Battiato, Devis Tuia Abstract Recent unsupervised domain adaptation (UDA) methods have shown great success in addressing classical domain shifts (e.g., synthetic-to-real), but they still suffer under complex shifts (e.g. geographical shift), where both the background and object appearances differ significantly across domains. Prior works showed that the language modality can help in the adaptation process, exhibiting more robustness to such complex shifts. In this paper, we introduce trust , a novel UDA approach that exploits the robustness of the language modality to guide the adaptation of a vision model. trust generates pseudo-labels for target samples from their captions and introduces a novel uncertainty estimation strategy that uses normalised CLIP similarity scores to estimate the uncertainty of the generated pseudo-labels. Such estimated uncertainty is then used to reweight the classification loss, mitigating the adverse effects of wrong pseudo-labels obtained from low-quality captions. To further increase the robustness of the vision model, we propose a multimodal soft-contrastive learning loss that aligns the vision and language feature spaces, by leveraging captions to guide the contrastive training of the vision model on target images. In our contrastive loss, each pair of images acts as both a positive and a negative pair and their feature representations are attracted and repulsed with a strength proportional to the similarity of their captions. This solution avoids the need for hardly determining positive and negative pairs, which is critical in the UDA setting. Our approach outperforms previous methods, setting the new state-of-the-art on classical (DomainNet) and complex (GeoNet) domain shifts. The code will be available upon acceptance. Figure 1: (a) geographical shifts strongly impact the appearance of both foreground objects and background. Contrarily, captions contain valuable information about the semantic class and they are minimally affected by the geographic shift, as stated in ( Kalluri, Majumder, and Chandraker 2024 ) . (b) Despite such apparent robustness, low quality captions may still lead to low classification accuracy. 1 Introduction Although deep learning models have achieved remarkable performance in lots of computer vision tasks, they still fall short in their ability to generalise to different domains. Retraining deep learning models on new data requires a big effort to acquire and manually label images and should be avoided. To address these limitations, Unsupervised Domain Adaptation (UDA) approaches for image classification have been proposed. UDA aims at transferring knowledge acquired on a labelled source domain to an unlabelled target domain, bridging the gap between them ( Xu et al. 2019 ; Wei et al. 2024 ; Sharma, Kalluri, and Chandraker 2021 ; Chen et al. 2022b ) . Recent UDA methodologies have been successful on classical domain shifts ( e.g. synthetic-to-real), but they suffer a lot on more complex shifts ( e.g. geographical), where both the background and objects’ appearance change between domains ( Kalluri, Xu, and Chandraker 2023 ) . Kalluri et al. ( Kalluri, Majumder, and Chandraker 2024 ) posited that, under complex shift, solely relying on images is challenging and they proposed LaGTran that integrates textual data for guiding the adaptation. Indeed, authors showed that textual data are more robust to complex shifts, as they semantically describe the image content instead of focusing on appearance details, which is essential for bridging the domain gap ( c.f. fig. 1 (a)). However, LaGTran ( Kalluri, Majumder, and Chandraker 2024 ) used textual data only for generating pseudo-labels for the target domain, highly underusing the potential of the language modality to reduce the domain shift. Moreover, they blindly rely on the pseudo-labels generated from the text, which may be incorrect due to two main factors: (a) the scarce quality of text descriptions, especially for crowd-sourced texts ( e.g. fig. 1 (b)); (b) the impact of domain shifts on language models, which is lower than image-based models but still present. For those reasons, solely training the vision model with pseudo-labels generated from the text is suboptimal for transferring the shift robustness from the language to the vision model. To overcome these limitations, we propose trust ( T ext R ob UST ness for unsupervised domain adaptation), a novel approach for UDA in image classification that exploits the potential of language guidance for the adaptation of a vision model. Similarly to LaGTran ( Kalluri, Majumder, and Chandraker 2024 ) , we use a language model to generate pseudo-labels on target samples from captions, but we also propose two novel components to cope with the shortcomings discussed above. First, to reduce the impact of incorrect pseudo-labels generated from low-quality captions, we propose a novel multimodal uncertainty estimation strategy that reweights the classification loss for target samples, by evaluating how well the captions semantically describe the target images. To this aim, we use a pretrained vision-language model ( i.e. CLIP ( Radford et al. 2021 ) ) to evaluate the semantic correlation between target images and their captions, which serves as a measure of the uncertainty/reliability of generated pseudo-labels. The estimated uncertainty is then used to reweight the contribution of the pseudo-labels in the classification loss, reducing the negative impact of wrong pseudo-labels obtained from low-quality captions. Second, to enhance the generalisation of the vision model, we aim to transfer the robustness to complex shifts from the language to the vision model, by promoting an effective alignment between their feature spaces. We propose to integrate a novel multimodal soft-contrastive learning loss, which uses the language modality to guide the contrastive training of the vision model. Unlike previous works, our soft-contrastive framework does not require to identify positive and negative pairs, as it assigns to each pair a score of “ positiveness ” and “ negativeness ” based on how likely they share the same semantic content. Then, a pair of images acts simultaneously as a positive and negative pair and attracts/repulses samples with a strength proportional to the similarity of their captions. The benefits of this strategy are multiple: (a) we encourage the vision model to match the language model’s feature space by attracting representations of images with similar captions and repulsing those with dissimilar ones; (b) differently than Chen et al. 2022a ; Litrico, Del Bue, and Morerio 2023 , we avoid contrasting images of the same class without relying on the output of the vision model, therefore limiting the confirmation bias; (c) we achieve a smoother contrastive training, where representations of each pair of samples are both attracted and repulsed based on how likely they share the same semantic content. We benchmark trust on three datasets representing classical (DomainNet, VisDA) and complex (GeoNet) shifts. On GeoNet, we obtain the best performance with a gain of +2.96\% to the best competitor; on DomainNet and VisDA, we improve performance of +2.53\% and +2.10\% , compared to state-of-the-art. To summarise, our main contributions are: • We introduce a novel uncertainty estimation strategy that leverages CLIP to evaluate the uncertainty of pseudo-labels, by measuring the semantic correlation between images and their captions. • We propose a novel multimodal soft-contrastive learning loss that uses the language modality to guide the contrastive training of the vision model. Our solution removes the problem of identifying positive and negative samples in UDA, preventing the confirmation bias. • We validate