KnowDR-REC: A Benchmark for Referring Expression Comprehension with Real-World Knowledge
Guanghao Jin, Jingpei Wu, Tianpei Guo, Yiyi Niu, Weidong Zhou, Guoyang Liu · 2025
Read KnowDR-REC: A Benchmark for Referring Expression Comprehension with Real-World Knowledge by Guanghao Jin, Jingpei Wu, Tianpei Guo, Yiyi Niu, Weidong Zhou, Guoyang Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14080.txt.
Introduction
Guanghao Jin and co-authors build a 2025 test for vision models that must find objects named in text. The task, called referring expression work, asks a model to locate the right item in one image. Older tests lean on clues inside the image alone. They thus fail to probe knowledge of the wider world.
Their benchmark, KnowDR-REC, needs fine reasoning across text and image with real-world facts. Tricky near-miss cases were built by small edits to the referring phrases. Those cases test strength against false but plausible answers. Three new scores trace how a model reasons, not just what it picks. Sixteen top models were tried on the set. All still struggled with knowledge-driven grounding. Many leaned on memorized shortcuts instead of true joint reasoning. The authors hope the test spurs more robust and clear systems.
Catalog record
| CREATOR | Guanghao Jin, Jingpei Wu, Tianpei Guo, Yiyi Niu, Weidong Zhou, Guoyang Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-14080
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
KnowDR-REC: A Benchmark for Referring Expression Comprehension with Real-World Knowledge Guanghao Jin, Jingpei Wu, Tianpei Guo, Yiyi Niu, Weidong Zhou, Guoyang Liu Abstract Referring Expression Comprehension (REC) is a popular multimodal task that aims to accurately detect target objects within a single image based on a given textual expression. However, due to the limitations of earlier models, traditional REC benchmarks either rely solely on intra-image cues or lack sufficiently fine-grained instance annotations, making them inadequate for evaluating the reasoning capabilities of Multi-modal Large Language Models (MLLMs). To address this gap, we propose a new benchmark, KnowDR-REC, characterized by three key features: Firstly, it is built upon real-world knowledge, requiring fine-grained multimodal reasoning across text and image. Secondly, the dataset includes elaborately constructed negative samples via fine-grained expression editing, designed to evaluate a model’s robustness and anti-hallucination ability. Lastly, we introduce three novel evaluation metrics to systematically explore the model’s internal reasoning process. We evaluate 16 state-of-the-art multimodal models on KnowDR-REC, with experimental results showing that existing MLLMs still struggle with knowledge-driven visual grounding tasks. Furthermore, we observe a decoupling between textual understanding and visual grounding in MLLMs, where many models are significantly influenced by memorized shortcut correlations, which severely affect their behavior on our benchmark and hinder genuine multimodal reasoning. We anticipate that the proposed benchmark will inspire future research towards developing more robust, interpretable, and knowledge-intensive visual grounding frameworks, driving the development of more reliable and robust multimodal systems for complex real-world scenarios. 1 1 footnotetext: Equal contribution. 2 2 footnotetext: Corresponding author. Figure 1: KnowDR-REC is a novel benchmark that aims to integrate external world knowledge into the textual expressions used in referring expression comprehension (REC). Code — https://github.com/LetItBe12345/KnowDR-REC Introduction The advent of Multi-modal Large Language Models (MLLMs) has significantly transformed artificial intelligence research, particularly impacting tasks that require joint vision-language understanding ( Liu et al. 2023 ; Huang et al. 2023 ; Zang et al. 2025 ) . Among these tasks, Referring Expression Comprehension (REC) ( Qiao, Deng, and Wu 2020 ) stands out as a core challenge, aiming to detect target objects in images based on natural language descriptions. Despite considerable progress, existing REC evaluation paradigms remain insufficient for adequately reflecting the inherent complexity of real-world applications. To bridge this gap, we introduce Knowledge-Driven Referring Expression Comprehension (KnowDR-REC), a novel benchmark explicitly designed to evaluate MLLMs on knowledge-intensive visual referring tasks. Representative examples from this benchmark are illustrated in Figure 1 . Traditional REC benchmarks predominantly focus on visual cues, spatial relations, and simple attribute matching within images ( Liu et al. 2019 ) . Widely-used benchmarks like RefCOCO ( Yu et al. 2016 ) , RefCOCO+ ( Yu et al. 2016 ) , and RefCOCOg ( Mao et al. 2016 ) have established foundational standards but significantly underestimate the complexity of natural human referential behaviors. Human references typically integrate external knowledge across temporal, cultural, and factual dimensions. For instance, a seemingly straightforward phrase like “the flag bearer wearing the Chinese delegation uniform from the 2024 Paris Olympics” actually requires deep chains of external knowledge, including temporal references (2024), events (Paris Olympics), national affiliation (Chinese delegation), and roles (flag bearer). Current REC benchmarks fail to cover such reasoning-intensive tasks, leaving a substantial gap in evaluating the ability of MLLMs to integrate external knowledge and perform multi-hop reasoning. Another major limitation of existing Referring Expression Comprehension (REC) benchmarks is the absence of fine-grained target annotations ( Wu et al. 2020 ) . Most current datasets provide only coarse-grained descriptions such as “the woman in red” or “the man wearing a hat,” which are insufficient for evaluating fine-grained perception in real-world scenarios. In practical scenarios, models must handle highly specific referring expressions, such as “the 2022 Nobel laureate in Physics”. These tasks not only challenge models’ fine-grained visual grounding capabilities but also test their capacity for cross-modal reasoning. Evidently, existing benchmarks fall short of adequately evaluating MLLM performance under such fine-grained conditions. Moreover, current REC benchmarks largely neglect unanswerable cases. While recent frameworks have introduced negative samples to assess refusal capabilities of models ( Liu et al. 2024c ; Schulter et al. 2023 ) , they often lack in-depth analysis of model errors. As a result, it remains difficult to evaluate a model’s robustness and anti-hallucination ability in ambiguous or invalid scenarios. The proposed KnowDR-REC benchmark advances REC evaluation along three key dimensions. First, we introduce external world knowledge, requiring models to perform multi-hop reasoning at both the textual and visual levels. In addition, we propose novel metrics designed to probe the models’ internal reasoning processes. Second, our benchmark emphasize fine-grained instance grounding, particularly targeting human subjects annotated with detailed attributes and contextual information. Third, we leverage temporal knowledge graphs ( Cai et al. 2022 ) to perform fine-grained editing of referring expressions, enabling diverse negative samples to assess model robustness and hallucination resistance. To validate the benchmark’s effectiveness, we conduct a comprehensive evaluation across 16 models, covering three categories: (1) closed-source generalist MLLMs; (2) open-source generalist MLLMs; and (3) open-source specialist MLLMs designed for visual grounding. Both conventional metrics and new metrics tailored for knowledge-intensive settings are employed. Experimental results reveal several critical issues with current multimodal models. Firstly, the models exhibit unsatisfactory performance on both positive and negative samples, indicating limitations in their textual grounding, visual grounding and anti-hallucination capabilities. Secondly, although textual reasoning accuracy is highly correlated with visual grounding correctness, models surprisingly demonstrate a certain level of grounding ability even when their textual reasoning is flawed. Most alarmingly, models often exhibit similar grounding behaviors regardless of the correctness of textual expressions, suggesting they rely not solely on the provided referring expressions but also heavily on memorized shortcuts. This phenomenon raises significant concerns about the reliability and robustness of these models in knowledge-intensive visual grounding tasks. Our main contributions are as follows: (1) We propose KnowDR-REC, the first benchmark explicitly designed to assess MLLMs on knowledge-driven, multi-hop reasoning in referring expression comprehension. (2) We develop a temporal knowledge graph-based negative sampling strategy to rigorously evaluate the robustness and anti-hallucination capabilities of MLLMs. (3) We introduce novel conditional evaluation metrics to precisely measure multimodal models’ textual reasoning and visual grounding performance. (4) Through extensive evaluations of 16 state-of-the-art MLLMs, we identify critical shortcomings of LLMs including reliance on memorized shortcuts and weak robustness against semantically subtle negative samples. Related Work Referring Expression Comprehension Referring Expression Comprehension (REC) is a core task in multimodal understanding that detects the visual target based on natural language. Classic benchmarks such as the RefCOCO series rely on short spatial expressions and have been nearly saturated—models like CogVLM ( Wang et al. 2024 ) achieve strong results even in zero-shot settings, with 92.44% on RefCOCO, 88.55% on RefCOCO+, and 90.67% on RefCOCOg ([email protected]), revealing a performance ceiling and limited room for further progress. To advance the task, recent datasets increase complexity: HC-RefLoCo ( Wei et al. 2024 ) uses long human-centric expressions, FineCops-Ref ( Liu et al. 2024c ) focuses on compositional reasoning, ReSeDis ( Huang, Zhang, and Satoh 2025 ) introduces cross-dataset and open-world settings, and MC-Bench ( Xu, Zhu, and Yang 2024 ) extends REC to the multi-image domain. Despite progress, existing benchmarks lack hard negatives, temporal cues, and knowledge-based