Dynamic Modality-Camera Invariant Clustering for Unsupervised Visible-Infrared Person Re-identification

Yiming Yang, Weipeng Hu, Haifeng Hu · 2024

Read Dynamic Modality-Camera Invariant Clustering for Unsupervised Visible-Infrared Person Re-identification by Yiming Yang, Weipeng Hu, Haifeng Hu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08231.txt.

Introduction

Yiming Yang, Weipeng Hu, and Haifeng Hu tackle person matching across normal and heat based cameras in a 2024 paper. The task is to tell when the same person shows up in both kinds of image, without labeled training data. Past tools grouped images by type and linked the groups, but they ignored gaps between cameras, which split one person into many groups.

Their framework joins three parts: one blends distance codes across image types and cameras, one refines groups with shifting search rules, and one trains on both single images and groups. In tests, the method came close to supervised tools, which need costly labels. The work shows that careful grouping can carry much of the load.

Catalog record

CREATORYiming Yang, Weipeng Hu, Haifeng Hu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-08231

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dynamic Modality-Camera Invariant Clustering for Unsupervised Visible-Infrared Person Re-identification Yiming Yang, Weipeng Hu, Haifeng Hu Abstract Unsupervised learning visible-infrared person re-identification (USL-VI-ReID) offers a more flexible and cost-effective alternative compared to supervised methods. This field has gained increasing attention due to its promising potential. Existing methods simply cluster modality-specific samples and employ strong association techniques to achieve instance-to-cluster or cluster-to-cluster cross-modality associations. However, they ignore cross-camera differences, leading to noticeable issues with excessive splitting of identities. Consequently, this undermines the accuracy and reliability of cross-modal associations. To address these issues, we propose a novel Dynamic Modality-Camera Invariant Clustering (DMIC) framework for USL-VI-ReID. Specifically, our DMIC naturally integrates Modality-Camera Invariant Expansion (MIE), Dynamic Neighborhood Clustering (DNC) and Hybrid Modality Contrastive Learning (HMCL) into a unified framework, which eliminates both the cross-modality and cross-camera discrepancies in clustering. MIE fuses inter-modal and inter-camera distance coding to bridge the gaps between modalities and cameras at the clustering level. DNC employs two dynamic search strategies to refine the network’s optimization objective, transitioning from improving discriminability to enhancing cross-modal and cross-camera generalizability. Moreover, HMCL is designed to optimize instance-level and cluster-level distributions. Memories for intra-modality and inter-modality training are updated using randomly selected samples, facilitating real-time exploration of modality-invariant representations. Extensive experiments have demonstrated that our DMIC addresses the limitations present in current clustering approaches and achieve competitive performance, which significantly reduces the performance gap with supervised methods. I Introduction Person re-identification is employed to identify and locate specific individuals among pedestrians captured in multiple camera surveillance scenarios [ 1 , 16 ] . This technology have been significantly developed during the last decade due to its important role in the fields of multimedia data retrieval and criminal investigation [ 3 , 4 , 5 ] . Earlier works on person Re-ID focus on the retrieval of pedestrian images captured by RGB cameras. However, such methods falter in low-light conditions. This is primarily due to the inherent limitation of RGB cameras in acquiring high-definition images in darkness. Therefore, visible-infrared person re-identification (VI-ReID) is proposed and employed to form 24-hour surveillance system, which aims to match infrared images under poor illumination with visible images under good illumination. The current VI-ReID methods focus on generation and subspace mapping technique to learn modality-invariant representations, achieving notable success [ 6 , 7 , 8 , 9 ] . However, their dependence on manually annotated associations between visible and infrared modalities can hinder the scalability and deployment of the VI-ReID model. Unsupervised Learning Visible-Infrared Person Re-Identification (USL-VI-ReID) is introduced to eliminate this reliance on annotations and gains increasing attention due to its promising potential. Fig. 1: Illustration of cross-modality and cross-camera discrepancies in clustering. Large variations caused by these discrepancies lead to identities splitting. Fine-tuning the network using these inaccurate labels obtains sub-optimal results. The USL-VI-ReID method explores cross-modal associations, eliminating the requirement for manual identity labeling [ 10 , 11 ] . Although DBSCAN [ 12 ] is an effective clustering algorithm for pseudo-labeling unlabeled data, challenges still arise due to substantial cross-modality and cross-camera variations within the images from the same ground truth ID [ 6 , 13 , 14 ] . Fig. 1 illustrates the challenge in clustering cross-modal person data. Variations across cameras and modalities lead to excessive identity splitting and hinder accurate label assignment. Fine-tuning the network using these labels may amplify the distances within the same class more than those between different classes. In this case, additional noise can be potentially introduced to adversely affect the performance of the model. Existing methods [ 15 , 16 , 17 , 18 , 19 ] primarily focus on eliminating differences between different modalities while neglecting the challenge of cross-camera discrepancy. For instance, previous approaches utilize graph matching [ 16 ] and optimal transmission [ 20 ] techniques to facilitate cross-modal cluster association. However, these methods are impeded by the severe issue of excessive identity splitting, which may affect the accuracy of the association. To address the above problems, we propose a novel Dynamic Modality-Camera Invariant Clustering (DMIC) framework to eliminate cross-modality and cross-camera discrepancies at the clustering level. The flowchart of DMIC is displayed in Fig. 2 . To be specific, DMIC naturally combines Modality-Camera Invariant Expansion (MIE), Dynamic Neighborhood Clustering (DNC) and Hybrid Modality Contrastive Learning (HMCL) into a joint framework. The MIE integrates inter-modal and inter-camera distance coding, generating robust embeddings for the clustering algorithm. This eradicates the implicit inclusion of modality and camera information in the distance coding, resulting in modality-camera invariant embeddings. Consequently, modality-camera invariant associations can be established. To tackle the issue of intra-class distances surpassing inter-class distances due to excessive identity splitting, the DNC employs two dynamic search strategies. Specifically, for the first stragegy, we dynamically narrow the search radius to include the reliable positive samples within clusters, thereby enhancing the model’s ability to distinguish between relevant and irrelevant samples. Subsequently, we dynamically broaden the search radius, using the model’s refined discrimination to effectively incorporate reliable cross-camera and cross-modality samples into the cluster. The second strategy involves recalibrating the expanded distance coding. This refinement facilitates a broaden affinities of more cross-camera instances, which can be leveraged to contribute to cross-camera invariant learning. Importantly, our strategies do not require additional parameters as a cost, effectively enhancing the model’s performance. Taking inspiration from [ 21 ] , we design HMCL to optimize instance-level and cluster-level distributions. We randomly select instance from different modalities as cluster’s centroid and update the representations of clusters in a real-time manner, which effectively reduces modal gap. To sum up, the main contributions of this paper are list as follows: • We propose a novel DMIC network for USL-VI-ReID that simultaneously eliminates both the cross-modality and cross-camera discrepancies in clustering. • The MIE fuses distance coding between inter-modal and inter-camera instances, which bridges cross-modality and cross-camera gaps. • The DNC consists of two dynamic search strategies that do not require additional parameters, facilitating early optimization of model discriminability and gradually extending generalization to different cameras and modalities. • The HMCL performs cluster-level and instance-level contrastive learning for intra-modality and inter-modality training. The cluster representations are updated using randomly selected cross-modal samples to obtain compact cross-modal distribution. • Extensive experiments on the SYSU-MM01 and RegDB datasets showcase the effectiveness and competitive performance of our DMIC framework. Fig. 2: The flowchart of Dynamic Modality-Camera Invariant Clustering (DMIC) model. Our DMIC model is composed of three key modules: Modality-Camera Invariant Expansion (MIE), Dynamic Neighborhood Clustering (DNC), and Hybrid Modality Contrastive Learning (HMCL). MIE fuses the distance encoding from multiple cameras to generate modality-camera invariant embeddings for clustering. DNC employs two dynamic search strategies that optimize the network’s performance, transitioning from improving discriminability to enhancing generalization. The estimated pseudo labels from MIE and DNC are used to initialize instance-level and cluster-level memories. HMCL includes intra-modality and inter-modality contrastive learning to learn modality-camera invariant representations. During the testing phase, our framework only utilizes the backbone for testing purposes. II Related work In this section, we provide a brief overview of the following areas: Supervised Visible-Infrared person ReID (SVI-ReID), Unsupervised Single-Modality Person ReID (USL-ReID), and Unsupervised Learning Visible-Infrared Person Re-Identification (USL-VI-ReID). II-A Supervised Visible-Infrared person ReID Supervised Visible-Infrared Person ReID (SVI-ReID) can be roughly divided

Read the complete text