ROGLE: Robust Global-Local Alignment with Automated Region Supervision for Text-Based Person Search
Chaodong Jia, Zequn Xie, Xibei Jia, Sihang Cai, Shulei Wang, Tao Jin · 2026
Read ROGLE: Robust Global-Local Alignment with Automated Region Supervision for Text-Based Person Search by Chaodong Jia, Zequn Xie, Xibei Jia, Sihang Cai, Shulei Wang, Tao Jin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01825.txt.
Introduction
Jia and co-authors improve search for a person from a text query. A user types a phrase about how a person looks. The system must find that person's photo. Models based on CLIP lean on whole-image traits. Short captions leave fine details sparse and weak.
Their ROGLE method joins global and local matching. It mines region-to-sentence pairs on its own. That step removes the need for costly hand labels. Global contrast training is fused with region-level fit. The team also builds the P-VLG benchmark data set. It holds over 100,000 marked regions with long captions. The authors report strong gains in fine-grained search.
Catalog record
| CREATOR | Chaodong Jia, Zequn Xie, Xibei Jia, Sihang Cai, Shulei Wang, Tao Jin |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01825
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ROGLE: Robust Global-Local Alignment with Automated Region Supervision for Text-Based Person Search Chaodong Jia, Zequn Xie, Xibei Jia, Sihang Cai, Shulei Wang, Tao Jin Abstract Text-Based Person Search (TBPS) aims to retrieve pedestrian images using natural language queries. However, existing TBPS models, especially those based on CLIP, struggle with fine-grained understanding due to global representational bias and semantic sparsity inherited from training on short captions. This results in weak fine-grained alignment, exacerbated by the scarcity of region-level annotations. To address this, we propose ROGLE (Robust Global-Local Embedding), a unified framework that overcomes reliance on costly manual annotations through an automated Region-to-Sentence Matching (RSM) strategy. RSM automatically mines pseudo region-sentence pairs for scalable fine-grained supervision. Furthermore, ROGLE employs a multi-granular learning strategy that fuses global contrastive learning with region-level local alignment. We also introduce the P-VLG Benchmark , a large-scale dataset constructed by curating and enriching images from established public benchmarks . It features over 100,000 annotated regions and rich long-form captions, making it the first TBPS benchmark to support both global and local assessment protocols. Extensive experiments show that ROGLE significantly outperforms existing approaches, particularly on challenging long-form queries. Code and the P-VLG benchmark will be made publicly available. 1 Introduction Text-Based Person Search (TBPS) is a challenging yet important retrieval task that aims to locate images of a target individual from a large gallery based solely on natural language descriptions Li et al. 2017 . Unlike traditional person re-identification methods that rely on example images He et al. 2021 ; Luo et al. 2019 , TBPS better models practical scenarios such as surveillance analysis or missing-person investigations, where no visual examples are available and verbal witness reports serve as the only source of information. Consequently, TBPS plays a vital role in enabling retrieval systems to operate effectively in open-world and zero-shot settings. Figure 1: Visualization of our Region-to-Sentence Matching (RSM) pipeline. The original person image (left) is paired with a long descriptive caption that is first decomposed into independent sentences. We apply SAM to segment the image into semantic regions (colored boxes) and compute cross-modal similarity between each region and each sentence. The highest-scoring sentence for each region is selected, producing five local region-sentence pairs on the right, corresponding to hairstyle, clothing, and footwear. Recent advances in TBPS have been largely driven by benchmark datasets such as CUHK-PEDES Li et al. 2017 , ICFG-PEDES Ding et al. 2021 , and RSTPReid Zhu et al. 2021 . Despite this progress, a key limitation persists: most existing systems, typically built upon large vision-language models such as CLIP Jiang and Ye 2023a ; Xie et al. 2025a ; Qin et al. 2024 , struggle with fine-grained understanding. Since these models are pre-trained on short, generic captions, they capture global semantics while neglecting local details. Consequently, training and evaluation datasets contain mainly brief, attribute-based annotations, offering only coarse-grained supervision that restricts a model’s ability to distinguish visually similar individuals or interpret complex queries. Moreover, natural language descriptions often include abstract relations or localized cues that do not directly correspond to visual regions, making fine-grained alignment challenging. The lack of region-level annotations further constrains learning meaningful part-level correspondences. To address the data-level bottleneck, we construct a large-scale benchmark named P-VLG using a novel Region-to-Sentence Matching (RSM) module. RSM decomposes long-form textual descriptions into atomic sentences and aligns them with semantically coherent image regions extracted by SAM Kirillov et al. 2023a . By leveraging CLIP Radford et al. 2021a embeddings to compute cross-modal similarity, RSM automatically mines pseudo region-sentence pairs, providing fine-grained supervision without manual annotation. The P-VLG benchmark overcomes the limitations of prior TBPS datasets, which often lack dense grounding and favor short captions. It contains 6,801 training identities, 48,485 images, and 68,990 enriched captions, with a test set of 400 unseen identities and 6,658 image-caption pairs. The validation set provides over 100,000 region-caption alignments, enabling evaluation of both global retrieval and region-level grounding. Based on these pseudo-labels, ROGLE adopts a dual-branch training paradigm. The first branch performs global contrastive learning to align whole-image and full-caption embeddings, while the second branch provides fine-grained supervision through multi-granular learning that integrates global and region-level local alignment. This design encourages the model to learn both contextualized global semantics and discriminative local features, essential for resolving ambiguous or compositional queries. In addition, a reliability-aware alignment calibration mechanism enhances robustness against noisy correspondences in large-scale data. Our contributions are summarized as follows: • We propose ROGLE, a unified global–local framework tackling semantic sparsity and misalignment. It incorporates an automated Region-to-Sentence Matching (RSM) module to mine pseudo-alignments from long captions, enabling fine-grained supervision without manual annotation. • We introduce the P-VLG Benchmark, the first dataset supporting multi-granular TBPS evaluation. It features enriched descriptions and over 100,000 region-level annotations to assess both holistic and localized alignment. • We design a multi-granular learning strategy synergizing global contrastive learning with region-level alignment. Extensive experiments demonstrate superior performance on challenging benchmarks, particularly for detailed and compositional queries. Figure 2: Visualization of the Region-to-Sentence Matching process in our P-VLG Benchmark. Given a pedestrian image and its long-form caption, SAM is used to generate region proposals \{I_{l,i}\}_{i=1}^{N} from the global image I_{g} . All image regions and textual sentences \{t_{l,j}^{cls}\}_{j=1}^{M} are encoded with CLIP, and cross-modal similarity is computed for all region-sentence pairs. The highest-scoring pair is then selected to create pseudo-aligned fine-grained supervision. 2 Related Work Figure 3: Overview of ROGLE. SAM segments the input image into local regions. We employ three textual supervisions: global long, concise short, and region-level captions. Dual encoders extract text ( \text{CLS}^{\text{long}}_{\text{text}},\text{CLS}^{\text{short}}_{\text{text}},\text{CLS}^{\text{region}}_{\text{text}} ) and image ( \text{CLS}^{\text{global}}_{\text{image}},\text{CLS}^{\text{local}}_{\text{image}} ) representations. A probabilistic module estimates pair reliability (clean/noisy/uncertain) to guide global ( \mathcal{L}_{\text{global}} ) and local ( \mathcal{L}_{\text{local}} ) alignment. 2.1 Text-Based Person Search Text-Based Person Search (TBPS) is a practical task that retrieves pedestrian images from large-scale galleries using free-form textual queries Li et al. 2017 . Early methods mainly relied on global contrastive learning to align whole-image and full-text representations Zhang and Lu 2018 ; Shu et al. 2022 . Recent advances have explored more sophisticated frameworks, such as agentic approaches for document-based retrieval Shu et al. 2026 and interaction-driven generation models Xie et al. 2025b . Additionally, newer studies have focused on query enhancement and context-aware representations Xie 2026 , as well as hierarchical visual perception to improve retrieval robustness Xie et al. 2026c . While effective for coarse semantic matching, global-only supervision struggles to capture subtle visual details, such as accessories, hairstyles, or multi-layered clothing, which are critical for distinguishing identities. This limitation motivates approaches that integrate both holistic and fine-grained alignment. 2.2 Fine-Grained Alignment To achieve finer alignment, existing strategies fall into two categories. Manual or part-based localization methods Gao et al. 2021 ; Wang et al. 2021 ; Shao et al. 2022 align phrases with corresponding image regions but rely heavily on labor-intensive annotations or specialized part-parsing modules, limiting scalability and generalization Zuo et al. 2024 . Detector-based supervision approaches, such as ViTAA Wang et al. 2020 , CLOC Chen et al. 2024 , and OWLv2 Minderer et al. 2023 , perform localized alignment via attribute modeling or open-vocabulary detection. Furthermore, research has extended into specialized domains, such as anomaly search utilizing cascade frameworks Xie et