Referring to Any Person
Qing Jiang, Lin Wu, Zhaoyang Zeng, Tianhe Ren, Yuda Xiong, Yihao Chen, Liu Qin, Lei Zhang · 2025
Read Referring to Any Person by Qing Jiang, Lin Wu, Zhaoyang Zeng, Tianhe Ren, Yuda Xiong, Yihao Chen, Liu Qin, Lei Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08507.txt.
Introduction
Jiang and seven co-authors redefine finding any person from a language description, in a paper from twenty twenty-five. Detecting a described individual has wide practical value. Existing models fail real-world tests and benchmarks assume one-to-one matching. Their HumanRef data set reflects messy real scenes instead.
Their RexSeek model joins a multimodal language model with a detector. Standard models ace older benchmarks yet stumble badly here. They cannot detect multiple described individuals reliably. RexSeek excels at human referring and generalizes to objects too. Five referable aspects and three task traits structure the redefinition.
Catalog record
| CREATOR | Qing Jiang, Lin Wu, Zhaoyang Zeng, Tianhe Ren, Yuda Xiong, Yihao Chen, Liu Qin, Lei Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08507
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Referring to Any Person Qing Jiang, Lin Wu, Zhaoyang Zeng, Tianhe Ren, Yuda Xiong, Yihao Chen, Liu Qin, Lei Zhang Abstract Humans are undoubtedly the most important participants in computer vision, and the ability to detect any individual given a natural language description, a task we define as referring to any person, holds substantial practical value. However, we find that existing models generally fail to achieve real-world usability, and current benchmarks are limited by their focus on one-to-one referring, that hinder progress in this area. In this work, we revisit this task from three critical perspectives: task definition, dataset design, and model architecture. We first identify five aspects of referable entities and three distinctive characteristics of this task. Next, we introduce HumanRef, a novel dataset designed to tackle these challenges and better reflect real-world applications. From a model design perspective, we integrate a multimodal large language model with an object detection framework, constructing a robust referring model named RexSeek. Experimental results reveal that state-of-the-art models, which perform well on commonly used benchmarks like RefCOCO/+/g, struggle with HumanRef due to their inability to detect multiple individuals. In contrast, RexSeek not only excels in human referring but also generalizes effectively to common object referring, making it broadly applicable across various perception tasks. Code is available at https://github.com/IDEA-Research/RexSeek Figure 1 : We introduce referring to any person, a task that requires detecting all individuals in an image which match a given natural language description, and a new model RexSeek designed for this task with strong perception and understanding capabilities that effectively captures attributes, spatial relations, interactions, reasoning, celebrity recognition, etc. 0 0 footnotetext: This work was done when Qing Jiang and Lin Wu were interns at IDEA. 1 1 footnotemark: 1 0 0 footnotetext: \dagger Corresponding author. 1 Introduction Humans are central to computer vision [ 30 , 63 , 12 , 16 , 28 , 27 , 10 , 46 , 11 , 65 , 45 , 4 , 21 , 85 , 84 ] , and the ability to identify and detect specific individuals based on natural language descriptions, a task we define as referring to any person, is crucial for numerous applications, including human-robot interaction, industrial automation, healthcare, etc. However, we argue that progress in this area has been hindered by unclear task definitions and a lack of high-quality data. Our findings show that despite achieving state-of-the-art performance on referring benchmarks RefCOCO/+/g [ 50 , 75 ] , most models remain impractical for real-world applications, as illustrated in Figure 2 . To address this challenge, we revisit this task from three perspectives: task definition, dataset construction, and model design. We begin by formally defining the task of referring to any person: given a natural language description and an input image, the model needs to detect all individuals in the image who match the description . To comprehensively capture the scope of this task, we identify five key aspects that define how humans can be referred to: i) Attributes: Encompassing intrinsic characteristics such as gender, age, action, clothing, etc. ii) Position: Describing spatial relationships both among individuals and between individuals and their surroundings. iii) Interaction: Accounting for human-to-human, human-to-object, and human-to-environment interactions. iv) Reasoning: Involving multi-step inference that considers multiple objects to resolve complex referrings. v) Celebrity Recognition: Identifying specific individuals, whether by their real names or characters names. Next, we identify three crucial characteristics that define this task: i) Multi-Instance Referring : A referring expression can correspond to multiple individuals. While mainstream referring datasets RefCOCO/+/g [ 50 , 75 ] typically assume that each expression refers to a single object, this does not align with real-world scenarios. We find through experiments that most models experience significant performance degradation when tasked with identifying more than one individual. ii) Multi-Instance Discrimination : The image should contain multiple individuals in addition to the target person. This setting ensures that the model fully comprehends the referring expression to identify the correct individual rather than simply detecting all people in the image. iii) Rejection of Non-existence : If the referred person is not present in the image, the model should refuse to generate a result rather than produce a hallucinated output. Based on the task definition, we manually constructed a novel dataset for human referring, named HumanRef. Unlike the traditional ReferItGame [ 29 ] annotation approach, where one annotator describes an object and another finds it based on the description, we adopt a different annotation methodology. Our process begins with annotators listing the key properties of individuals in an image according to the predefined referable entities. Next, for each person, they determine whether these properties apply and result in a property dictionary. Finally, a large language model [ 71 ] composes these properties into referring expressions. HumanRef comprises 103,028 referring statements, with each expression referring to an average of 2.2 instances. We also split a benchmark from HumanRef with 6,000 referring expressions spanning six subsets, ensuring comprehensive coverage across all referable properties. Figure 2 : Visualization results of Qwen2.5-VL [ 3 ] , InternVL-2.5 [ 14 ] , and DeepSeek-VL2 [ 70 ] on the human referring task. Despite achieving strong results on referring benchmarks RefCOCO/+/g [ 50 , 75 ] , state-of-the-art models struggle when tasked with identifying multiple individuals as they output an insufficient number of bounding boxes. From the model design perspective, we argue that a robust referring model should possess two key characteristics: i) Robust Perception Ability : The model should be capable of detecting all individuals in an image. ii) Strong Language Comprehension : The model should effectively interpret complex language descriptions of people. To address these requirements, we introduce RexSeek, a detection-oriented multimodal large language model specifically designed for this task. Inspired by ChatRex [ 25 ] , we formulate referring as a retrieval-based task. RexSeek lintegrates a person detector [ 60 ] as its box input, ensuring strong perception capabilities while incorporating Qwen2.5 [ 71 ] as the LLM to enhance language comprehension. We adopt a multi-stage training approach that progressively refines both detection and comprehension skills, equipping RexSeek with strong referring capabilities. Experimental results indicate that most state-of-the-art models [ 74 , 49 , 9 , 3 , 70 , 14 , 60 , 25 ] exhibit performance degradation on the HumanRef benchmark, despite achieving strong results on RefCOCO/+/g. The primary limitation is that these models typically detect only a single instance, as they are trained on datasets that assume one-to-one referring. In contrast, RexSeek, trained on HumanRef, exhibits strong referring capabilities. Additionally, benefiting from the multi-stage training approach, RexSeek also emerges with the ability to refer to generalized objects, extending its applicability beyond human-centric tasks. To summarize, our contributions are threefold: • We introduce referring to any person with a clear definition by identifying five aspects of referable entities and three key characteristics that distinguish this task. • We introduce HumanRef, a novel referring dataset, and establish a challenging benchmark to drive progress in human-centric referring expression research. • We propose RexSeek, a detection-oriented multimodal large language model trained through a multi-stage process, demonstrating strong referring capabilities for both humans and general objects. 2 Related Work Referring Expression Comprehension Task. Referring Expression Comprehension (REC) [ 56