VG-CoT: Towards Trustworthy Visual Reasoning via Grounded Chain-of-Thought

2026

Read VG-CoT: Towards Trustworthy Visual Reasoning via Grounded Chain-of-Thought (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-21396.txt.

Introduction

Byeonggeuk Lim and three colleagues make vision-language reasoning trustworthy by tying each reasoning step to real visual evidence. Their VG-CoT dataset links steps to image regions through a fully automated three-stage pipeline. Detection and OCR extract object- and text-level evidence, GPT-4o generates grounded step-by-step reasoning, and rationale-driven open-set detection refines the grounding. A new benchmark scores rationale quality, answer accuracy, and reasoning-answer alignment. Tests on LLaVA-1.5 and Qwen2-VL show consistent gains on most metrics.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31625
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.21396

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VG-CoT: Towards Trustworthy Visual Reasoning via Grounded Chain-of-Thought Abstract The advancement of Large Vision-Language Models (LVLMs) requires precise local region-based reasoning that faithfully grounds the model’s logic in actual visual evidence. However, existing datasets face limitations in scalability due to extensive manual annotation and lack of explicit alignment between multi-step reasoning and corresponding image regions, which constrains the evaluation of model trustworthiness. To address these challenges, we propose the Visual Grounding Chain-of-Thought (VG-CoT) dataset, which explicitly links each reasoning step to real visual evidence within the image through a fully automated three-stage pipeline. The pipeline first extracts object- and text-level visual evidence using state-of-the-art detection and OCR models, then generates step-by-step grounded reasoning with GPT-4o, and finally refines the grounding through a rationale-driven open-set detection process. In addition, we introduce a new benchmark that comprehensively evaluates LVLMs reasoning across three complementary dimensions: Rationale Quality, Answer Accuracy, and Reasoning-Answer Alignment. Experiments with representative LVLMs, including LLaVA-1.5 and Qwen2-VL, demonstrate consistent improvements on most evaluation metrics, confirming that VG-CoT effectively enhances trustworthy, evidence-based reasoning while maintaining scalable and cost-efficient dataset construction. The dataset and code will be released publicly upon acceptance to facilitate further research. Keywords: LVLMs Reasoning, Visual Grounded Rationale Dataset, Reasoning Evaluation VG-CoT: Towards Trustworthy Visual Reasoning via Grounded Chain-of-Thought Byeonggeuk Lim 1 , Kyeonghyun Kim 2 , JungMin Yun 2 and YoungBin Kim 1, 2 Graduate School of Advanced Imaging Science, Multimedia & Film, Chung-Ang University 1 Department of Artificial Intelligence, Chung-Ang University 2 {banggeuk, khyun8072, cocoro357, ybkim85}@cau.ac.kr Abstract content 1. Introduction The development of Large Language Models (LLMs) has recently led to the rise of Large Vision-Language Models (LVLMs), which simultaneously understand visual and linguistic information 34 ; 31 . LVLMs have demonstrated outstanding performance in comprehensive, image-level understanding, and the focus of research is gradually expanding toward the ability for precise understanding of local regions within an image 9 ; 19 ; 36 ; 29 . Specifically, the ability to accurately identify specific regions within an image and interpret their spatial and semantic relationships is emerging as a core requirement for performing advanced vision-language tasks such as autonomous driving, robotics, and medical image analysis 16 ; 38 ; 17 . However, this local region-based reasoning ability is highly dependent on the quality and composition of the training dataset 30 ; 5 . Currently, most widely used vision-language datasets primarily focus on evaluating comprehensive, image-level understanding, which limits models from learning fine-grained reasoning based on the detailed attributes of individual objects or the relationships between objects 7 ; 35 ; 3 ; 13 . Although some studies have proposed datasets that include local region information, they still rely on large-scale manual annotation, making it difficult to avoid the constraints of cost and scalability 32 ; 33 ; 15 . This manual construction method presents a more pronounced limitation, especially in tasks that require complex visual relationships or multi-step reasoning. This suggests the need for a dataset that can support the learning of complex, local region-based reasoning in a scalable and efficient manner. Existing multimodal Chain-of-Thought (CoT) datasets designed to support multi-step reasoning suffer from a structural limitation: the reasoning process is not explicitly linked to specific visual evidence within the actual image 21 ; 32 . In other words, because the location information (spatial coordinates) of the objects or text within the image that serve as the rationale for the inference is not provided, it is difficult to verify which areas of the image the model’s reasoning was based on. This absence of visual evidence leads to a structural limitation in model evaluation regarding whether the model derived the correct answer using the right evidence as its rationale, causing existing benchmarks to only assess the accuracy of the final answer 30 ; 5 . Therefore, a dataset where the entire process of reasoning is explicitly aligned with real visual evidence is essential for verifying whether the logic presented by the model is faithful to the actual image evidence 5 ; 24 . This also provides the foundation for a new benchmark capable of systematically measuring not only the accuracy of the final answer but also the accuracy of the rationale and the logical soundness of the reasoning process. Based on this necessity, this study proposes the Visual Grounding Chain-of-Thought (VG-CoT) dataset. VG-CoT is a dataset constructed by explicitly aligning visual evidence from an image with the reasoning process across various tasks, achieving both efficiency and scalability through an automated three-stage pipeline. (i) First, initial visual evidence is extracted using state-of-the-art object detection and OCR models. (ii) Based on this, step-by-step reasoning is generated using GPT-4o. (iii) Finally, using the generated reasoning process as a clue, visual evidence is more precisely captured through a text-based object detection model. This automated pipeline effectively solves the cost issue associated with manual annotation while contributing to the improvement of the quality of the reasoning process and the reliability of rationale alignment. Furthermore, this study proposes a new benchmark that goes beyond the existing evaluation methods centered on answer accuracy. This benchmark comprehensively measures Rationale Quality, Answer Accuracy, and Reasoning-Answer Alignment, which represents the relationship between the two metrics. Specifically, it utilizes GPT-4o 1 as an evaluator to precisely assess the reasoning process using detailed metrics: logical coherence, reasoning completeness, and visual evidence utilization. This allows for an in-depth verification of whether the model has reasoned logically based on credible visual evidence, going beyond simply checking if the model provided the correct answer. The key contributions of this study are summarized as follows: 1. We propose the VG-CoT dataset, which explicitly aligns the reasoning process with real visual evidence within the image through an automated three-stage pipeline. 2. We establish a new benchmark that comprehensively measures Rationale Quality and Reasoning–Answer Alignment, going beyond simple answer accuracy. 3. We evaluate representative LVLMs using VG-CoT to analyze their capability in utilizing visual evidence, providing insights into the future direction of LVLMs research. 2. Related Work Figure 1: Examples of the Proposed VG-CoT Dataset across Three Task Types. 2.1. Datasets for LVLMs Reasoning Early datasets for LVLMs primarily focused on image-wide understanding, which limited their ability to train models for fine-grained, local region-based reasoning 18 ; 3 ; 13 . To overcome this limitation, datasets that explicitly include local region information were proposed, but most relied on manual annotation, a method that requires immense cost and time, thereby restricting data diversity and scalability 15 ; 23 ; 32 . Consequently, there emerged a need for datasets that overcome the limitations of manual annotation and require scalable yet detailed visual understanding. In line with this necessity, studies have been conducted to encourage models to perform complex reasoning processes 27 ; 21 ; 6 ; 22 . ScienceQA 21 aimed to enhance model explainability by providing rationales for questions, and subsequently, M3CoT 6 pointed out that ScienceQA was limited to a specific domain and single-step reasoning. M3CoT was thus designed to necessarily require multi-step reasoning across multiple domains, increasing the complexity of inference. These studies marked significant progress in that they explicitly attempted to show the reasoning process through multimodal CoT. However, a limitation shared by both studies is the lack of an explicit link between each reasoning

Read the complete text