Multilingual Text-to-Image Person Retrieval via Bidirectional Relation Reasoning and Aligning

Min Cao, Xinyu Zhou, Ding Jiang, Bo Du, Mang Ye · 2025

Read Multilingual Text-to-Image Person Retrieval via Bidirectional Relation Reasoning and Aligning by Min Cao, Xinyu Zhou, Ding Jiang, Bo Du, Mang Ye (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-17685.txt.

Introduction

Min Cao, with Xinyu Zhou and four co-authors, extends person search by text beyond English in a 2025 arXiv paper. The task is finding a specific person's photo from a written description. Mixing language and images is hard because the two forms carry information differently. Prior methods were English centric and often missed fine details or needed preset part alignments.

The team built a multilingual benchmark, first translating with large language models, then refining with domain knowledge. Their framework, called Bi-IRRA, learns alignment across both languages and modalities. One module predicts masked image and text parts in both directions, exposing local links. Another module aligns global features across several dimensions. The method set new best results on all the multilingual test sets.

Catalog record

CREATORMin Cao, Xinyu Zhou, Ding Jiang, Bo Du, Mang Ye
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-17685

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multilingual Text-to-Image Person Retrieval via Bidirectional Relation Reasoning and Aligning Thanks: This work is supported by the National Natural Science Foundation of China under Grants 62476188 and 62176188, the Natural Science Foundation of the Jiangsu Higher Education Institutions of China, Key Laboratory of New Generation Artificial Intelligence Technology & Its Interdisciplinary Applications (Southeast University), Ministry of Education, China (Corresponding author: Mang Ye). Min Cao, Xinyu Zhou, and Min Zhang are with the School of Computer Science and Technology, Soochow University, SuZhou 215000, China (e-mail: [email protected], [email protected]). Ding Jiang, Bo Du, and Mang Ye are with the School of Computer Science, Wuhan University, Wuhan 430072, China (e-mail: [email protected]). Min Cao, Xinyu Zhou, Ding Jiang, Bo Du, Mang Ye Abstract Text-to-image person retrieval (TIPR) aims to identify the target person using textual descriptions, facing challenge in modality heterogeneity. Prior works have attempted to address it by developing cross-modal global or local alignment strategies. However, global methods typically overlook fine-grained cross-modal differences, whereas local methods require prior information to explore explicit part alignments. Additionally, current methods are English-centric, restricting their application in multilingual contexts. To alleviate these issues, we pioneer a multilingual TIPR task by developing a multilingual TIPR benchmark, for which we leverage large language models for initial translations and refine them by integrating domain-specific knowledge. Correspondingly, we propose Bi-IRRA: a Bidirectional Implicit Relation Reasoning and Aligning framework to learn alignment across languages and modalities. Within Bi-IRRA, a bidirectional implicit relation reasoning module enables bidirectional prediction of masked image and text, implicitly enhancing the modeling of local relations across languages and modalities, a multi-dimensional global alignment module is integrated to bridge the modality heterogeneity. The proposed method achieves new state-of-the-art results on all multilingual TIPR datasets. Data and code are presented in https://github.com/Flame-Chasers/Bi-IRRA. I Introduction Given a text query, Text-to-Image Person Retrieval (TIPR) [ 1 ] aims to identify the most relevant person images from an extensive gallery of such images. The task is similar to the person re-identification task (Re-ID) [ 2 , 3 , 4 ] , which involves identifying person images across cameras based on the image query. In contrast to the structured image query in Re-ID, the text query in TIPR takes the form of free, flexible characters, making it more accessible and offering substantial application potential in public safety domains. In recent years, TIPR has garnered increasing attention [ 5 , 6 , 7 , 8 , 9 ] . A key challenge in TIPR is the inherent modality gap between vision and language, driving research toward robust cross-modal alignment. These efforts can be broadly categorized into global-matching [ 10 , 11 , 12 , 13 ] [ 14 , 15 , 16 ] and local-matching methods [ 17 , 18 , 19 , 20 ] [ 21 ] . The former aligns global text-image representations at the coarse-grained level via cross-modal matching loss functions (Fig. 1 (a)), while the latter establishes fine-grained associations between textual entities and image body parts (Fig. 1 (b)). Despite notable progress in this task, two critical issues remain to be addressed. The first issue is the limited focus on language environments. Current methods [ 8 , 5 , 6 , 22 , 7 ] center around addressing TIPR with English as the default text query. However, in practical application, there is a growing need for supporting multiple languages, ( e.g. , Chinese and French) as queries for TIPR. Biased towards a single language ( i.e. , English), current methods struggle to achieve accurate retrieval when confronted with diverse language requirements in real-world scenarios. Another issue involves the constraint of existing cross-modal alignment strategies. Global-matching methods align the global representations of texts and images, often overlooking the need to bridge modality heterogeneity at a more detailed, fine-grained level, potentially impacting performance. On the other hand, local-matching methods are tailored to build the explicit correspondence between body parts and textual descriptions with the aid of external technologies and predefined rules. As a result, they confine the exploration of cross-modal alignment within the boundaries of these set rules, and also pose resource-intensive demands as it necessitates the extraction and storage of multiple local part representations of images and texts during inference. In this paper, we pioneer a multilingual TIPR task. Specifically, we build the multilingual TIPR benchmark and propose Bi-IRRA: a cross-modal Bidirectional Implicit Relation Reasoning and Aligning framework to learn alignment across languages and modalities at both coarse-grained and fine-grained levels. This work is centered on addressing both data and framework aspects. Fig. 1: Illustration of different TIPR methods. (a) Global-matching methods directly align global image and text feature representations. (b) Local-matching methods explicitly extract and align local image and text representations. (c) Our bidirectional implicit relation reasoning and aligning method not only implicitly reasons about the relations among all local tokens but also aligns global image and text representations in a multilingual environment. Data . For a novel multilingual TIPR task, a critical obstacle is the scarcity of multilingual TIPR data to support its research. While manual annotation of multilingual TIPR data presents a direct solution, it is labor-intensive and impractical for covering a wide range of languages over extensive image data. An alternative solution involves utilizing Large Models (LMs) [ 23 , 24 , 25 , 26 , 27 , 28 ] for automatic translation on existing TIPR datasets, thereby extending them beyond English. However, directly using LMs for translation often introduces noise due to their lack of domain-specific knowledge. For this, we develop a LMs-driven Domain Adaptive Translation (LDAT) pipeline, consisting of translation, filtering, and rewriting phases. After an initial translation by Large Language Models (LLMs) [ 29 , 23 , 24 ] in the translation phase, we identify clean and noisy translation texts in the filtering phase. Subsequently, the clean texts with corresponding person images serve as the supervision signal to finetune Multimodal Large Language Models (MLLMs) [ 25 , 26 , 27 ] . The finetuned MLLMs, enriched with comprehensive domain-specific knowledge, are employed to rewrite noisy texts in the rewriting phase, thus effectively mitigating the noise issue. The proposed LDAT, as a concise translation pipeline, enables the cost-effective acquisition of the high-quality multilingual TIPR benchmark. Framework . In contrast to the traditional TIPR task that deals solely with heterogeneity between text and image modalities, the multilingual TIPR encapsulates modality heterogeneity and linguistic diversity challenges. In response, we propose the Bi-IRRA framework, designed to establish robust global alignment and explore implicit fine-grained relations across diverse languages and modalities (as depicted in Fig. 1 (c)). Specifically, Bi-IRRA comprises a Bidirectional Implicit Relation Reasoning (Bi-IRR) module and a Multi-dimensional Global Alignment (Md-GA) module. The Bi-IRR module performs bidirectional prediction of masked text and image, enabling the implicit modeling of local relations between vision and language. It includes a bi-lingual Masked Language Modeling (MLM) pretext task and a cross-lingual Distillation Masked Image Modeling (D-MIM) pretext task, tailored for adaptive modeling and interaction across languages. Meanwhile, Md-GA aligns global text and image representations from multiple dimensions, facilitating the extraction of discriminative global text and image representations. The Md-GA module comprises a bi-lingual Image-Text Contrastive (ITC) pretext task and a bi-lingual Asymmetric Image-Text Matching (A-ITM) pretext task, with the former applied to the

Read the complete text