Towards Understanding Visual Grounding in Vision-Language Models
Georgios Pantazopoulos, Eda B. Özyiğit · 2025
Read Towards Understanding Visual Grounding in Vision-Language Models by Georgios Pantazopoulos, Eda B. Özyiğit (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-10345.txt.
Introduction
Pointing at what words describe is the skill Georgios Pantazopoulos and Eda B. Özyiğit survey in this 2025 paper on visual grounding in vision-language models. Grounding means finding the region of an image or video that matches a phrase, a talent behind question answering, detailed captioning, and robot control.
The survey maps how modern grounded models are built, what parts they combine, and where they are used, along with the benchmarks and metrics that grade them. It also traces the links between grounding, step-by-step multimodal reasoning, and chain-of-thought methods. The authors close by sorting the open difficulties, from fine spatial precision to trustworthy evaluation, and naming the research directions they judge most promising.
Catalog record
| CREATOR | Georgios Pantazopoulos, Eda B. Özyiğit |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 84 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-10345
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Understanding Visual Grounding in Vision-Language Models Georgios Pantazopoulos, Eda B. Özyiğit Abstract Visual grounding refers to the ability of a model to identify a region within some visual input that matches a textual description. Consequently, a model equipped with visual grounding capabilities can target a wide range of applications in various domains, including referring expression comprehension, answering questions pertinent to fine-grained details in images or videos, captioning visual context by explicitly referring to entities, as well as low- and high-level control in simulated and real environments. In this survey paper, we review representative works across the key areas of research on modern general-purpose vision-language models (VLMs). We first outline the importance of grounding in VLMs, then delineate the core components of the contemporary paradigm for developing grounded models, and examine their practical applications, including benchmarks and evaluation metrics for grounded multimodal generation. We also discuss the multifaceted interrelations among visual grounding, multimodal chain-of-thought, and reasoning in VLMs. Finally, we analyse the challenges inherent to visual grounding and suggest promising directions for future research. K eywords Visual Grounding, Vision-Language Models, Multimodal Grounded Generation 1 Introduction Visual grounding li2etal025 represents a fundamental challenge in multimodal artificial intelligence, requiring systems to establish precise correspondences between natural language descriptions and specific regions or objects within visual scenes. This capability is essential for enabling machines to comprehend and manipulate visual content through linguistic interaction, forming the cornerstone of human-computer interfaces that rely on natural language communication about visual information. The field encompasses a diverse spectrum of tasks and applications, ranging from core problems such as referring expression comprehension ( kazemzadeh-etal-2014-referitgame ; he2023grec ; krishna2017visual ) and grounded captioning ( PontTuset_eccv2020 ; peng2023kosmos ) to complex applications including grounded visual question answering ( zhu2016visual7w ; lai2024lisa ) , low- and high-level robotic control in interactive environments ( shridhar2020alfred ; jiang2022vima ; gao2023alexa ) , and agents operating across web, mobile, and desktop environments ( cheng2024seeclick ; gou2025navigating ) . Visual grounding has evolved through several distinct paradigms. Early approaches combined Convolutional Neural Networks (CNNs) with Recurrent Neural Networks (RNNs) ( Mao_2016_CVPR ; Liu_2017_ICCV ; yu2016modeling ) . These were followed by specialised transformer-based models emerging from multi-task vision language pre-training (VLP) ( kamath2021mdetr ; deng2021transvg ; chen2020uniter ; wang2022ofa ; cho2021unifying ; pantazopoulos-etal-2023-multitask ) , and more recently by methods that leverage vision-language models (VLMs) ( youferret ; chen2023shikra ; bai2023qwen ; bai2025qwen2 ) . This progression toward grounded text generation has enabled multimodal models to establish intrinsic connections between linguistic expressions and corresponding visual elements, equipping machines with increasingly sophisticated fine-grained multimodal comprehension capabilities. This survey primarily examines multimodal grounded generation in VLMs published from 2022 onward, while also incorporating key insights from foundational VLP models. The main contributions of this work are: (i) a comprehensive overview of grounding in VLMs, extending beyond the traditional focus on generalised Referring Expression Comprehension (REC) ( kazemzadeh2014referitgame ) to encompass underexplored research domains; (ii) an analysis of how core architectural components influence grounding capabilities, offering a perspective absent from prior surveys ( liang2024survey ; wu2023multimodal ; huang2023visual ; caffagni2024revolution ) ; (iii) a coverage of additional grounding-related tasks, such as grounded captioning, grounded visual question answering, and agents interacting with Graphical User Interfaces (GUIs); and (iv) a discussion of grounding as a component of multimodal reasoning, both as an intermediate step and as part of solution-level reasoning processes. The remainder of this paper is organised as follows. Section 2 presents the essential background, including definitions and relevant research domains. Section 3 synthesises prior work on the development of VLMs with grounding capabilities. Section 4 provides a comprehensive analysis of existing applications, datasets, and benchmarks, spanning image-level visual grounding to agents that interact with environments and web interfaces. Finally, Section 5 discusses the key components involved in developing grounded VLMs and examines the challenges and open questions facing the next generation of these models. 2 Preliminaries 2.1 What is Visual Grounding? In earlier works, visual grounding, where a model is tasked with localising a specific region within a visual context that aligns with an input textual query ( xiao2024towards ) , is often referred to as REC ( kazemzadeh2014referitgame ) or phrase localisation (PL) ( plummer2015flickr30k ) , where the input query describes exactly one entity in an image. However, this definition is narrow and does not adequately encompass neighbouring fields such as grounded captioning, visual question answering grounded in visual elements, and interactive agent-based environments. In these settings, the textual query may take various forms, for instance, specifying a region of interest (e.g. describe the region with a concise caption), posing a question (e.g. which of these objects is used to cut items?), or giving an instruction (e.g. clean up old cookies from the Amazon store). Accordingly, we define visual grounding more broadly as the capability to precisely localise and identify specific objects, regions, or concepts within a visual context, conditioned on natural language descriptions or instructions, thereby bridging textual understanding and visual reasoning. 2.2 The Role of Grounding in Modern VLMs Grounding in VLMs is essential not only for enabling fine-grained downstream applications, but also for establishing an interpretable link between predicted perceptions and actual localisation within the visual context. Although VLM outputs often imply an understanding of visual elements, grounding explicitly verifies whether the model can accurately identify and localise the relevant objects, regions, or concepts. This connection between language and precise visual evidence is critical for both transparency and reliability in multimodal reasoning. Prior research indicates that hallucination in VLMs arises from an excessive reliance on the language prior, with the dependence on the visual prompt diminishing as more tokens are generated ( favero2024multi ; goyal2017making ; wang2023evaluation ; frank-etal-2021-vision ; lin2023revisiting ) . Without proper grounding, these models may generate plausible-sounding descriptions that do not correspond to actual visual content, resulting in unreliable outputs in critical applications. Grounding addresses this issue by ensuring that when a model describes an object or relationship, it can demonstrate its understanding by precisely identifying where that information appears in the visual input. Consequently, VLMs without grounding capabilities often struggle to establish precise correspondences between textual descriptions and visual elements. This can result in responses that are semantically plausible but spatially inaccurate. Recent research ( youferret ; chen2023shikra ) suggests that grounding mitigates this issue by encouraging models to commit to specific spatial locations, thereby enhancing both the accuracy and interpretability of their visual understanding. However, subsequent studies ( lin2023revisiting ) have shown that grounding objectives do not exhibit a causal relationship with reductions in object hallucinations, as quantified in visual question answering and image captioning tasks, indicating that further investigation is necessary. Nevertheless, the ability to precisely refer to regions that align with open-ended textual input remains an invaluable asset, enabling VLMs to support a wide range of downstream applications ( kazemzadeh-etal-2014-referitgame ; zhu2016visual7w ; zheng2024seeact ; wu2023webui ) . 2.3 Representing Visual References Visual grounding requires converting continuous spatial coordinates into formats that language models can process effectively. Consequently, the design choice of representing regions plays a crucial role in model development. There are two broad categories for representing the bounding box: object-centric and pixel-level. Object-centric Object-centric methods present the model with region proposals, optionally accompanied