KARL: Knowledge-Aware Reasoning and Reinforcement Learning for Knowledge-Intensive Visual Grounding
Xinyu Ma , Ziyang Ding, Zhicong Luo, Chi Chen, Zonghao Guo, Xuebo Liu, Derek F. Wong , Zhen Zhao, Xiaoyi Feng, Maosong Sun · 2025
Read KARL: Knowledge-Aware Reasoning and Reinforcement Learning for Knowledge-Intensive Visual Grounding by Xinyu Ma , Ziyang Ding, Zhicong Luo, Chi Chen, Zonghao Guo, Xuebo Liu, Derek F. Wong , Zhen Zhao, Xiaoyi Feng, Maosong Sun (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-12797.txt.
Introduction
Xinyu Ma, Ziyang Ding, Zhicong Luo, and seven colleagues tackle knowledge-intensive visual grounding in this two thousand twenty-five paper. The task asks a model to locate an object named by a fine-grained, domain-specific term rather than a generic description. Multimodal large language models know many entities and ground generic phrases well, yet often fail to use that knowledge for specialized concepts. The authors call this the knowledge-grounding gap.
Their remedy first builds knowledge-guided reasoning data that pushes models to activate relevant entity knowledge while grounding. Then KARL, a reinforcement learning framework, adjusts reward signals according to how well the model has mastered each entity. The team also releases KVG-Bench, spanning ten domains with over a thousand curated test cases. Experiments showed consistent gains over baselines and stronger generalization to unseen categories.
Catalog record
| CREATOR | Xinyu Ma , Ziyang Ding, Zhicong Luo, Chi Chen, Zonghao Guo, Xuebo Liu, Derek F. Wong , Zhen Zhao, Xiaoyi Feng, Maosong Sun |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-12797
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
KARL: Knowledge-Aware Reasoning and Reinforcement Learning for Knowledge-Intensive Visual Grounding Xinyu Ma , Ziyang Ding, Zhicong Luo, Chi Chen, Zonghao Guo, Xuebo Liu, Derek F. Wong , Zhen Zhao, Xiaoyi Feng, Maosong Sun Abstract Knowledge-Intensive Visual Grounding (KVG) requires models to localize objects using fine-grained, domain-specific entity names rather than generic referring expressions. Although Multimodal Large Language Models (MLLMs) possess rich entity knowledge and strong generic grounding capabilities, they often fail to effectively utilize such knowledge when grounding specialized concepts, revealing a knowledge– grounding gap between internal knowledge and grounding predictions. To address this challenge, we propose a knowledge-aware training paradigm for KVG. Our approach first constructs knowledge-guided reasoning data to encourage models to activate domain-relevant entity knowledge during grounding, and then introduces KARL, a Knowledge-Aware Reinforcement Learning framework that adaptively modulates reward signals according to the model’s estimated knowledge mastery of different entities. To facilitate systematic evaluation, we introduce KVG-Bench, a benchmark spanning 10 domains with 1.3K curated test cases covering 531 images and 882 entities. Extensive experiments show that our approach consistently outperforms a wide range of baseline models and achieves substantially stronger cross-domain generalization on unseen categories. The data, codes, and models are released at https://github.com/thunlp/KARL 1 Introduction Human experts demonstrate fine-grained discrimination of visual concepts by flexibly leveraging domain knowledge to refine discriminative features, resulting in superior domain-specific visual perception compared to non-experts [ 36 , 15 , 11 ] . In contrast, recent Multimodal Large Language Models (MLLMs) exhibit rich fine-grained knowledge of diverse entities [ 14 , 13 ] and strong generic visual grounding capabilities [ 40 , 6 , 1 ] , yet often fail to operationalize such knowledge when perception must culminate in a grounding prediction. As illustrated in fig. 1 a, for the same entity, a model may correctly identify the queried concept yet predict an erroneous bounding box when required to ground it in the image. This discrepancy suggests the presence of a knowledge–grounding gap, where internal entity knowledge and grounding ability do not reliably compose into accurate knowledge-intensive grounding. To systematically investigate this issue, we introduce Knowledge-Intensive Visual Grounding (KVG), a visual grounding task that extends conventional visual grounding [ 16 , 49 ] by requiring both fine-grained visual perception and domain-specific entity knowledge integration. As shown in fig. 1 a, each query in KVG utilizes domain-specific terminology (“Clumber Spaniel”) rather than generic descriptions (“the left dog”), and images contain visually similar distractors that necessitate knowledge-guided differentiation. Thus, successful grounding depends not only on recognizing visual patterns, but also on aligning domain-level knowledge with spatial evidence selection. Unlike the mathematical and geometric tasks studied by previous reasoning MLLMs [ 44 , 45 ] , KVG emphasizes the integration of entity knowledge within visual perception processes. By requiring the composition of entity knowledge with fine-grained visual discrimination, KVG poses substantial challenges for current MLLMs. For instance, although Qwen3-VL-8B [ 1 ] achieves high accuracy on the widely used RefCOCO benchmark [ 49 ] , its performance substantially declines on KVG ( fig. 1 b), indicating that strong generic grounding does not readily generalize to knowledge-intensive settings. Moreover, even when equipped with explicit step-by-step reasoning prompting or trained with recent reasoning-guided visual grounding frameworks [ 2 , 48 ] , models fail to demonstrate consistent performance improvements on KVG. These observations suggest that the core difficulty lies not in insufficient reasoning depth, but in the ineffective alignment between fine-grained entity knowledge and grounding decisions. Figure 1 : (a) While the MLLM can correctly recognize the entity (Q1), it fails to ground it (Q2), revealing an inconsistency between knowledge and grounding. Our method integrates knowledge-guided reasoning to bridge this gap. (b) (b) KARL achieves substantially stronger grounding performance than the baseline model and zero-shot CoT prompting, showing that knowledge-guided reasoning for KVG cannot be effectively induced by simple prompting alone. To mitigate this limitation, we propose a knowledge-aware training paradigm tailored to knowledge-intensive visual grounding. Our approach begins with constructing knowledge-guided reasoning data that encourages models to explicitly activate domain-relevant entity knowledge prior to producing grounding predictions. By guiding models to articulate and connect entity-level knowledge with visual evidence, this stage aims to strengthen the alignment between knowledge utilization and grounding. Building on this foundation, we introduce KARL, a Knowledge-Aware Reinforcement Learning framework that dynamically modulates optimization signals at the entity level. Rather than applying uniform reward signals across all samples, KARL adjusts reinforcement strength according to the model’s estimated knowledge mastery of different entities, enabling differentiated optimization across heterogeneous entity types. This design is motivated by our observation that entities vary substantially in their prior knowledge availability within MLLMs, leading to uneven grounding behavior across entities. By explicitly coupling entity knowledge utilization with visual grounding optimization, our method aims to reduce the knowledge–grounding gap and enhance the model’s ability to leverage entity knowledge during grounding. To facilitate systematic evaluation on KVG, we introduce KVG-Bench, a high-quality, expertly-curated benchmark encompassing 10 distinct domains with 1.3K manually curated test cases linked to 531 images and 882 entities. Our extensive experiments conducted across KVG and related tasks reveal two key findings: (1) Knowledge-guided reasoning training consistently improves KVG performance over direct fine-tuning and generic reasoning strategies, demonstrating the effectiveness of explicitly activating entity-level knowledge during grounding; (2) KARL achieves substantially stronger cross-domain generalization on unseen categories, particularly compared with standard GRPO optimization, indicating that knowledge-aware reward scaling helps better generalize across entities with different knowledge levels. In summary, our contributions are threefold: • We introduce Knowledge-Intensive Visual Grounding (KVG) task and KVG-Bench, a benchmark designed to evaluate models’ ability to leverage domain-specific entity knowledge for visual grounding. Our empirical observations suggest the presence of a knowledge–grounding gap in current MLLMs. • We propose a knowledge-guided reasoning training strategy that constructs CoT reasoning data to encourage models to explicitly activate and align entity-level knowledge with visual evidence during grounding, differing from recent reasoning-guided grounding approaches that primarily emphasize structured reasoning depth. • We present KARL, a Knowledge-Aware Reinforcement Learning framework that dynamically modulates optimization signals according to entity-level knowledge mastery rather than applying uniform reward schemes. This design promotes more balanced optimization across entities with heterogeneous knowledge levels and leads to improved generalization in knowledge-intensive grounding. 2 Related Work 2.1 Multimodal Large Language Models Recent years have witnessed rapid advancements in MLLMs [ 31 , 23 , 22 , 6 , 46 , 1 ] , demonstrating strong performance in tasks such as visual grounding [ 5 , 32 , 47 , 26 , 20 ] and reasoning [ 35 , 44 , 10 , 51 ] . Early MLLM-based grounding approaches such as Shikra [ 5 ] , Groma [ 26 ] , and LISA [ 20 ] adopt end-to-end architectures that align textual queries with image regions, enabling direct prediction of bounding boxes or segmentation masks. While effective, these methods primarily rely on learned visual-text alignment and typically perform grounding in a single-step manner without explicit reasoning. Inspired by recent reinforcement learning approaches for enhancing reasoning in large language models [ 12 ] , recent works such as UniVG-R1 [ 2 ] , Perception-R1 [ 48 ] , and Visual-RFT [ 24 ] adopt reinforcement learning to improve visual grounding performance by strengthening