Enhancing Visual Representation for Text-based Person Searching Journal: Journal of L a T e X Templates

Wei Shen, Ming Fang, Yuxia Wang, Jiafeng Xiao, Diping Li, Huangqun Chen, Ling Xu, Weifeng Zhang · 2024

Read Enhancing Visual Representation for Text-based Person Searching Journal: Journal of L a T e X Templates by Wei Shen, Ming Fang, Yuxia Wang, Jiafeng Xiao, Diping Li, Huangqun Chen, Ling Xu, Weifeng Zhang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-20646.txt.

Description

Abstract Text-based person search aims to retrieve the matched pedestrians from a large-scale image database according to the text description. The core difficulty of this task is how to extract effective details from pedestrian images and texts, and achieve cross-modal alignment in a common latent space.

Catalog record

CREATORWei Shen, Ming Fang, Yuxia Wang, Jiafeng Xiao, Diping Li, Huangqun Chen, Ling Xu, Weifeng Zhang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-20646

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Visual Representation for Text-based Person Searching Journal: Journal of L a T e X Templates Wei Shen, Ming Fang, Yuxia Wang, Jiafeng Xiao, Diping Li, Huangqun Chen, Ling Xu, Weifeng Zhang Abstract Text-based person search aims to retrieve the matched pedestrians from a large-scale image database according to the text description. The core difficulty of this task is how to extract effective details from pedestrian images and texts, and achieve cross-modal alignment in a common latent space. Prior works adopt image and text encoders pre-trained on unimodal data to extract global and local features from image and text respectively, and then global-local alignment is achieved explicitly. However, these approaches still lack the ability of understanding visual details, and the retrieval accuracy is still limited by identity confusion. In order to alleviate the above problems, we rethink the importance of visual features for text-based person search, and propose VFE-TPS , a V isual F eature E nhanced T ext-based P erson S earch model. It introduces a pre-trained multimodal backbone CLIP to learn basic multimodal features and constructs Text Guided Masked Image Modeling task to enhance the model’s ability of learning local visual details without explicit annotation. In addition, we design Identity Supervised Global Visual Feature Calibration task to guide the model learn identity-aware global visual features. The key finding of our study is that, with the help of our proposed auxiliary tasks, the knowledge embedded in the pre-trained CLIP model can be successfully adapted to text-based person search task, and the model’s visual understanding ability is significantly enhanced. Experimental results on three benchmarks demonstrate that our proposed model exceeds the existing approaches, and the Rank-1 accuracy is significantly improved with a notable margin of about 1 % ∼ 9 % 1\%\sim 9\% . Our code can be found at https://github.com/zhangweifeng1218/VFE_TPS . 1 Introduction (a) (b) (c) Figure 1: (a) Person identity confusion: The pedestrians with same ID have lower similarity than pedestrians with different IDs. The bar chart below each image represents its visual feature. (b) Existing global-local alignment approaches, which needs explicitly conduct global and local alignment for image and text. (c) Our visual feature enhancement aided matching paradigm, which introduces auxiliary tasks to enhance visual features and only global alignment is needed. The dashed arrows and boxes are only active during training. Text-based person search [ 1 ] aims to retrieve the corresponding pedestrian images according to the text query which describes the attributes of the target person. It is a comprehensive application combining cross-modal retrieval [ 2 ] , person re-identification [ 3 ] and visual tracking [ 4 , 5 , 6 , 7 ] . Different from traditional person re-identification task which uses images as query, text-based person search retrieves target pedestrian images based on text description, which is much more simple and intuitive. For example, when conducting a suspect search task, witnesses can only provide a text description of the suspect rather then a specific images. Therefore, text-based person search has better practical value, especially in the field of intelligent security. In the recent decay, text-based person search has become one of the hottest research topics in multimodal intelligence [ 8 ] , and has gained widespread attention from academia and industry. While significant progress has been made in recent years, text-based person search task remains a challenging task. This is primarily due to: (1) The task of text-based person search is a typical cross-modal task, where the model needs to simultaneously understand the text query and the content of pedestrian images, extract appropriate features from both text and images, and then project them into a common latent space to achieve cross-modal alignment [ 9 ] . (2) In contrast to typical cross-modal retrieval tasks involving natural scene images, text-based person search encounters the challenge of subtle differences among person images, which leads to a common issue of person identity confusion [ 10 ] . As shown in Figure 1 (a), the left and middle images share the same person identity, yet their similarity is only 0.34, which is much lower than the similarity between the middle image and the right image which have different identities. This is mainly due to the fact that pedestrian images are often captured by different surveillance cameras at different times and locations, leading to variations in lighting, camera angles, and other environmental factors. The key to addressing the above problem lies in the algorithm’s ability to extract detailed information, which is closely relevant to the current task, from the images. For example, in Figure 1 (a), the model needs to focus on details such as the color of the pedestrian’s shoes and the presence of a backpack, rather than paying attention to factors like the pose of the pedestrian. Early solutions [ 11 , 12 , 13 ] rely on global feature alignment, which typically involve using convolutional neural networks and recurrent neural networks to extract global features from images and texts respectively, and then projecting them into a common latent space for alignment. These methods overlook local detailed features and have limited cross-modal interaction capabilities, resulting in low retrieval accuracy. Recognizing the significance of local features in text-based person search task, latter works [ 14 , 15 , 16 , 17 ] adopt global-local alignment method, which further explore the extraction and alignment of local features on top of global alignment. As shown in Figure 1 (b), such methods typically utilize statistical priors for pedestrian images [ 18 ] or employ pre-trained segmentation models [ 19 ] to explicitly extract features of different image parts. Simultaneously, a natural language grammar parser [ 20 ] is used to parse the text query into phrases. Subsequently, alignment of image parts and text phrases can be conducted. While these global-local alignment approaches have made significant progress in retrieval accuracy compared to global alignment models, this explicit alignment of local features from images and texts is constrained by the capability of segmentation models and grammar parsers. This limitation results in inaccurate extraction of local features and aligning all local features individually during the inference stage leads to excessive computational costs. Furthermore, both the aforementioned global alignment and global-local alignment methods utilize image encoder and text encoder pre-trained on unimodal data, lacking the capability of multimodal feature learning. In this paper, we propose a Visual Feature Enhanced Text-based Person Search (VFE-TPS) model, which is illustrated in Figure 1 (c). In order to extract rich cross-modal features from pedestrian images and text queries, we no longer rely on encoders pre-trained on unimodal data. Instead, we initialize our image and text encoders using Contrastive Language-Image Pre-training (CLIP) [ 21 , 22 ] model which is pre-trained on a large-scale image-text dataset. Existing works have demonstrated CLIP’s outstanding generalization and zero-shot capability in tasks such as image classification and video understanding [ 23 ] . However, the training images used by CLIP are natural scene images collected from the internet, which exhibit significant differences in feature distribution when compared to pedestrian images. Therefore, prior studies [ 24 , 25 ] have indicated that directly transferring knowledge from CLIP to text-based person search task by freezing the model parameters is unfeasible. Therefore, instead of freezing

Read the complete text