Beyond Pedestrians: Caption-Guided CLIP Framework for High-Difficulty Video-based Person Re-Identification
Shogo Hamano, Shunya Wakasugi, Tatsuhito Sato, Sayaka Nakamura · 2026
Read Beyond Pedestrians: Caption-Guided CLIP Framework for High-Difficulty Video-based Person Re-Identification by Shogo Hamano, Shunya Wakasugi, Tatsuhito Sato, Sayaka Nakamura (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-07740.txt.
Introduction
Hamano and three colleagues build CG-CLIP for matching people across video cameras in hard settings like sports and dance. Similar clothes and fast motion defeat many current systems. Captions from multimodal language models refine identity features through a memory step. Fixed-length learnable tokens then gather space-and-time features at low cost. The system tops current leaders on MARS and iLIDS-VID and on two new hard sets for sports and dance.
Catalog record
| CREATOR | Shogo Hamano, Shunya Wakasugi, Tatsuhito Sato, Sayaka Nakamura |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52016 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.07740
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Pedestrians: Caption-Guided CLIP Framework for High-Difficulty Video-based Person Re-Identification Shogo Hamano, Shunya Wakasugi, Tatsuhito Sato, Sayaka Nakamura Abstract In recent years, video-based person Re-Identification (ReID) has gained attention for its ability to leverage spatiotemporal cues to match individuals across non-overlapping cameras. However, current methods struggle with high-difficulty scenarios, such as sports and dance performances, where multiple individuals wear similar clothing while performing dynamic movements. To overcome these challenges, we propose CG-CLIP, a novel caption-guided CLIP framework that leverages explicit textual descriptions and learnable tokens. Our method introduces two key components: Caption-guided Memory Refinement (CMR) and Token-based Feature Extraction (TFE). CMR utilizes captions generated by Multi-modal Large Language Models (MLLMs) to refine identity-specific features, capturing fine-grained details. TFE employs a cross-attention mechanism with fixed-length learnable tokens to efficiently aggregate spatiotemporal features, reducing computational overhead. We evaluate our approach on two standard datasets (MARS and iLIDS-VID) and two newly constructed high-difficulty datasets (SportsVReID and DanceVReID). Experimental results demonstrate that our method outperforms current state-of-the-art approaches, achieving significant improvements across all benchmarks. 1 Introduction Video-based person Re-Identification (ReID) 33 ; 36 ; 61 , which aims to identify individuals across video sequences, has attracted significant attention over the past decade. Numerous methods have been developed to address this task, including approaches based on CNNs 8 ; 10 ; 13 ; 53 and Transformers 3 ; 37 ; 43 ; 57 . Recent works 30 ; 59 ; 55 ; 26 have explored leveraging large-scale pre-trained Vision-Language Models (VLMs), such as Contrastive Language-Image Pre-training (CLIP) 40 , to enhance performance. For example, CLIP-ReID 30 applies CLIP to image-based person ReID by optimizing text tokens through prompt learning. TF-CLIP 55 introduces a one-stage text-free approach for video-based person ReID, where identity-specific image features (CLIP-Memory) are computed from all training data and used as a substitute for text features. Figure 1 : Two basketball players with similar appearances. They wear identical uniforms and shoes of similar colors but differ in their hairstyles and jersey numbers. These subtle differences can be explicitly represented in textual descriptions. Figure 2 : Comparison of existing approaches and our method. (a) Prompt learning and text-free learning approaches. (b) Our proposed CG-CLIP framework that leverages text captions. Despite these advancements, most existing methods focus on relatively straightforward scenarios, such as videos of pedestrians captured by surveillance cameras. In contrast, real-world applications of person ReID, such as tracking 11 ; 60 —where person ReID is crucial for long-term identity association—often involve more complex scenarios including sports or dance performances 24 ; 41 , where individuals are dressed in similar uniforms or costumes and move rapidly. When faced with such scenarios, current methods often struggle due to two main challenges: (i) capturing fine-grained discriminative features and (ii) efficiently aggregating spatiotemporal information. Figure 1 depicts an example of two visually similar individuals, along with textual descriptions highlighting their unique characteristics. At first glance, it is challenging to distinguish them; however, close inspection reveals subtle differences in hairstyles and jersey numbers. By representing these attributes in textual form, nuanced differences that are often overlooked become more prominent and easier to identify. This observation underscores the potential of textual information as a complementary resource for enhancing person ReID performance in high-difficulty domains. Despite this, most existing methods do not explicitly utilize such descriptions and instead opt for using learnable tokens or image features, as shown in Fig. 2 (a). These approaches make it difficult to capture fine-grained distinctions that appear in only small regions of the image. The effective exploitation of robust spatiotemporal features also plays a crucial role in video-based person ReID. Previous methods 55 ; 53 ; 47 employ self-attention mechanisms to aggregate information across temporal and spatial dimensions. However, this approach has a drawback in terms of computational cost, which scales quadratically with the input token length. This issue becomes particularly evident in dynamic scenes, where increased motion often requires higher frame rates or higher-resolution images, substantially increasing memory and computational costs. To address these issues, we propose CG-CLIP, a novel caption-guided CLIP framework that leverages explicit textual descriptions and learnable tokens to capture robust representations. Our method consists of two key components: Caption-guided Memory Refinement (CMR) and Token-based Feature Extraction (TFE). We first synthesize training captions that describe subtle attributes ( e.g . , hairstyle, jersey number, and shoes) using Multi-modal Large Language Models (MLLMs) 1 . Next, as illustrated in Fig. 2 (b), we encode all captions with a frozen CLIP text encoder and build an identity-specific Text Memory ( i.e . , one text prototype per identity) by averaging caption embeddings. Unlike instance-wise image–text alignment, CMR uses the Text Memory as queries to selectively attend to identity-discriminative patch tokens from the training batch. It then refines the contrastive memory target by injecting caption-guided fine-grained cues into the image-based CLIP-Memory. Captions are not required during inference, which maintains the efficiency of the proposed method. Furthermore, we propose the TFE module for effective extraction of spatiotemporal information. In the TFE module, a cross-attention mechanism with learnable tokens is applied sequentially along the temporal and spatial dimensions to all tokens output by the image encoder. This design keeps the attention cost linear in the number of input tokens, making it more scalable when denser temporal sampling or higher-resolution inputs are needed in fast-motion scenarios. We evaluate our approach on two standard benchmarks, MARS and iLIDS-VID. Additionally, we propose two high-difficulty datasets, named SportsVReID and DanceVReID. These datasets are generated from Multi-Object Tracking (MOT) datasets (SportsMOT 9 and DanceTrack 42 ), containing scenes of sports and dance performances. Experiments on those datasets demonstrate the performance improvements achieved by our method. In summary, our main contributions are as follows: • We propose CG-CLIP, a novel caption-guided CLIP framework with Caption-guided Memory Refinement (CMR), which is capable of extracting fine-grained features using explicit captions. • We develop a Token-based Feature Extraction (TFE) module to integrate spatiotemporal features while keeping computational overhead low. • Extensive experiments demonstrate that our method consistently outperforms state-of-the-art approaches on both existing benchmarks and our newly constructed datasets. 2 Related work 2.1 Person ReID datasets The person ReID task can be categorized into image-based person ReID 18 ; 54 ; 58 ; 66 and video-based person ReID 8 ; 43 ; 59 ; 62 . In both categories, most existing datasets focus on pedestrian scenarios. For image-based person ReID, popular datasets include Market-1501 63 , MSMT17 48 , and DukeMTMC 64 . In recent years, datasets containing sports scenes have also been proposed, such as the Player Re-Identification dataset 44 and SoccerNet 14 , which consist of basketball and soccer scenes, respectively. In the case of video-based person ReID, widely used benchmarks include PRID 21 , iLIDS-VID 45 , MARS 62 , and LS-VID 27 . In contrast to image-based person ReID, there are currently few datasets comprising scenes such as sports or dance performances that contain multiple individuals with similar appearances. To address this gap, we introduce two new video-based person ReID datasets that specifically target such high-difficulty settings. 2.2 Video-based person ReID methods Video-based person ReID methods aim to utilize both spatial and temporal cues to extract discriminative features from videos of persons. Early studies have explored various approaches, including