CLIP4VI-ReID: Learning Modality-shared Representations via CLIP Semantic Bridge for Visible-Infrared Person Re-identification

Xiaomei Yang, Xizhan Gao, Sijie Niu, Fa Zhu, Guang Feng, Xiaofeng Qu, and David Camacho · 2025

Read CLIP4VI-ReID: Learning Modality-shared Representations via CLIP Semantic Bridge for Visible-Infrared Person Re-identification by Xiaomei Yang, Xizhan Gao, Sijie Niu, Fa Zhu, Guang Feng, Xiaofeng Qu, and David Camacho (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-10309.txt.

Introduction

Xiaomei Yang and six co-authors match people across visible-light and infrared cameras, a re-identification task where the two image types differ wildly in character. Their network, CLIP4VI-ReID, uses text as a bridge: a generation module writes identity-related text semantics for visible images only, aligning visible and text features first. An embedding module then corrects infrared features using that text, injecting identity semantics into the shared image encoder.

A final alignment stage refines the text so it carries only identity information, sharpening cross-modal matching and the distinctiveness of the shared representations. Because the infrared side aligns to visible images indirectly through language, the physical gap between spectra is sidestepped. Experiments on widely used visible-infrared re-identification datasets beat prior state-of-the-art methods.

Catalog record

CREATORXiaomei Yang, Xizhan Gao, Sijie Niu, Fa Zhu, Guang Feng, Xiaofeng Qu, and David Camacho
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-10309

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CLIP4VI-ReID: Learning Modality-shared Representations via CLIP Semantic Bridge for Visible-Infrared Person Re-identification Xiaomei Yang, Xizhan Gao, Sijie Niu, Fa Zhu, Guang Feng, Xiaofeng Qu, and David Camacho Abstract The core of visible-infrared person re-identification (VI-ReID) lies in learning modality-shared representations across different modalities. Given the significant modality gap between visible and infrared images, VI-ReID still remains a challenge. Recent research tends to use pre-trained CLIP to extract modality-shared representations, but they fail to consider differences in physical characteristics of natural images and infrared images when extracting visual features of infrared images and generating their text semantics. This causes the learned infrared text semantics to struggle to delicately describe the images, thereby limiting the performance of VI-ReID. In addition, after obtaining the text semantics, they do not make any adaptive adjustments, resulting in an increased risk of mismatches. To address these issues, we propose a novel CLIP-driven modality-shared representation learning network named CLIP4VI-ReID for VI-ReID task, which consists of Text Semantic Generation (TSG), Infrared Feature Embedding (IFE), and High-level Semantic Alignment (HSA). Specifically, considering the huge gap in the physical characteristics between natural images and infrared images, the TSG is designed to generate text semantics only for visible images, thereby enabling preliminary visible-text modality alignment. Then, the IFE is proposed to rectify the feature embeddings of infrared images using the generated text semantics. This process injects id-related semantics into the shared image encoder, enhancing its adaptability to the infrared modality. Besides, with text serving as a bridge, it enables indirect visible-infrared modality alignment. Finally, the HSA is established to refine the high-level semantic alignment. This process ensures that the fine-tuned text semantics only contain id-related information, thereby achieving more accurate cross-modal alignment and enhancing the discriminability of the learned modal-shared representations. Extensive experimental results demonstrate that the proposed CLIP4VI-ReID achieves superior performance than other state-of-the-art methods on some widely used VI-ReID datasets. The code will be available at https://github.com/xxxx(anonymous)/CLIP4VI-ReID . I Introduction Fig. 1: The motivation of the paper: (a) Visible (RGB) images and infrared (IR) images share the same high-level semantics. Therefore, the text semantics can serve as a bridge to mitigate the modality gap. (b) Existing single-modality methods struggle to handle VI-ReID tasks and exhibit weak feature representation for infrared images. Besides, due to the huge gap between natural images and infrared images, existing cross-modality methods struggle to capture the fine-grained details of infrared objects, and may introduce semantic noise, thereby affecting their performance. Furthermore, they did not adaptively adjust the obtained text semantics. (c) In contrast, our CLIP4VI-ReID model proposes a coarse-to-fine RGB-IR cross-modal alignment scheme, i.e., it first generates text semantics for visible images, then uses the text semantics as a bridge to correct the embedding of infrared images. Finally, it adaptively adjusts the obtained text semantics and refines the high-level semantic alignment among the three modalities. Person re-identification (ReID) is a critical computer vision and computational intelligence task aimed at retrieving query objects from large-scale gallery databases captured by non-overlapping cameras [ 1 , 2 ] . This task plays an essential role in enhancing public security, for instance, with the aid of intelligence surveillance systems, ReID enables the tracking of criminal suspects and the locating of missing persons. In recent years, driven by the increasing demand for computational intelligence and behavior analysis, this field has garnered significant global attention and achieved remarkable advancements in computer vision research. However, most existing ReID models focus on visible-visible person re-identification, which performs effectively under favorable lighting conditions [ 3 , 2 , 4 , 5 , 6 ] . Nevertheless, their performance often deteriorates under low-light or nighttime conditions, limiting their applicability in modern systems requiring all-weather monitoring. Infrared (IR) cameras offer a viable solution by capturing thermal imaging information even in complete darkness, providing an alternative that is not constrained by ambient light. Given that most modern social surveillance systems are equipped with dual-mode cameras capable of automatically switching modes based on environmental lighting, it is imperative to investigate visible-infrared cross-modal person ReID (VI-ReID) technology. VI-ReID enables pedestrian matching between visible and infrared images across varying lighting conditions, thereby supporting 24/7 monitoring requirements. Different from single-modal person ReID, VI-ReID focuses on retrieving images from a gallery of infrared or visible images that match the identity of a given visible or infrared query. In this case, the modalities of the gallery and query are different. As a result, researchers encounter significant challenges, including inter-modality differences and intra-modality variations. Inter-modality differences are primarily manifested in the substantial discrepancies between visible (RGB) and IR images, which encompass aspects such as color, texture, and brightness. RGB images are distinguished by their rich color information, whereas infrared images predominantly capture thermal radiation data. This inherent distinction introduces additional complexity to the person re-identification task [ 7 ] . Intra-modality variations refers to the appearance differences among images of the same person captured under the same modality at different times, locations, or camera viewpoints. These differences can stem from various factors, such as changes in illumination, occlusion, pose variation, inconsistent resolution, complex backgrounds, which may become even more pronounced in a cross-modal tasks. To address these challenges, researchers have proposed many cross-modal VI-ReID methods [ 8 ] , which can be grouped into modality-compensated methods [ 9 , 10 , 11 , 12 ] and modality-shared methods [ 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 ] . Modality-compensated methods first use some generative models such as GAN [ 22 , 23 ] to generate missing modalities from existing modalities to reduce inter-modality differences. Then, they extract discriminative features from the original and generated modalities to deal with the intra-modality variation problem. However, the presence of noise interference hinders the training stability, making it challenging to ensure the quality of generated images [ 24 ] . On the contrary, modality-shared methods first map two modality images into a shared feature space, and then, they use some loss functions [ 25 ] or decoupling modules [ 26 ] to enhance the discriminative ability of shared features. Although existing methods have shown certain effectiveness in using solely images to train models, such manner leads to the lack of high-level semantic information of the visual content learned by the model [ 27 ] . Recently, the visual language learning paradigm has attracted much attention for its powerful ability to extract semantically rich visual features. The representative Contrastive Language Image Pre-training (CLIP) model [ 28 ] has achieved remarkable success in numerous downstream vision tasks [ 29 , 30 ] . And it has also been successfully used in the person Re-ID field, including single-modality ReID methods [ 31 , 27 ] and cross-modality ReID methods [ 24 , 32 ] . As shown in the upper left part of Fig. 1 (b), single-modality ReID methods (such as CLIP-ReID [ 27 ] ) transfer CLIP to the image- or video-based person ReID tasks and show that combining visual information with corresponding linguistic description through CLIP can make the model perceive high-level semantic information related to the target pedestrian. However, these methods are designed for RGB data and cannot effectively handle cross-modality ReID tasks. Therefore, how to apply CLIP

Read the complete text