Argus : Vision-Centric Reasoning with Grounded Chain-of-Thought
Yunze Man, De-An Huang, Guilin Liu, Shiwei Sheng, Shilong Liu, Liang-Yan Gui, Jan Kautz, Yu-Xiong Wang, Zhiding Yu, [0.25cm] University of Illinois Urbana-Champaign NVIDIA · 2025
Read Argus : Vision-Centric Reasoning with Grounded Chain-of-Thought by Yunze Man, De-An Huang, Guilin Liu, Shiwei Sheng, Shilong Liu, Liang-Yan Gui, Jan Kautz, Yu-Xiong Wang, Zhiding Yu, [0.25cm] University of Illinois Urbana-Champaign NVIDIA (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23766.txt.
Introduction
Yunze Man and co-authors give a vision-language model eyes that check their work. Multimodal models reason fluently about images yet often fumble tasks where the crux is exactly where to look. Their 2025 paper, Argus, inserts object grounding into the reasoning itself. The model names and boxes the region each step of its chain of thought depends on. Attention becomes an explicit, inspectable signal.
This grounded chain of thought steers the model's visual focus toward the goal as the answer develops. On varied benchmarks, Argus excelled both at multimodal reasoning and at the referring-expression task of finding the object a phrase describes. Breakdowns of the design confirm the grounding steps earn their keep. The authors argue that multimodal intelligence should be built vision-first: language may conduct, but the evidence is in the pixels, and the model should be made to point at it.
Catalog record
| CREATOR | Yunze Man, De-An Huang, Guilin Liu, Shiwei Sheng, Shilong Liu, Liang-Yan Gui, Jan Kautz, Yu-Xiong Wang, Zhiding Yu, [0.25cm] University of Illinois Urbana-Champaign NVIDIA |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23766
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Argus : Vision-Centric Reasoning with Grounded Chain-of-Thought Yunze Man, De-An Huang, Guilin Liu, Shiwei Sheng, Shilong Liu, Liang-Yan Gui, Jan Kautz, Yu-Xiong Wang, Zhiding Yu, [0.25cm] University of Illinois Urbana-Champaign NVIDIA Abstract Recent advances in multimodal large language models (MLLMs) have demonstrated remarkable capabilities in vision-language tasks, yet they often struggle with vision-centric scenarios where precise visual focus is needed for accurate reasoning. In this paper, we introduce Argus to address these limitations with a new visual attention grounding mechanism. Our approach employs object-centric grounding as visual chain-of-thought signals, enabling more effective goal-conditioned visual attention during multimodal reasoning tasks. Evaluations on diverse benchmarks demonstrate that Argus excels in both multimodal reasoning tasks and referring object grounding tasks. Extensive analysis further validates various design choices of Argus, and reveals the effectiveness of explicit language-guided visual region-of-interest engagement in MLLMs, highlighting the importance of advancing multimodal intelligence from a visual-centric perspective. Project page: https://yunzeman.github.io/argus/ . Figure 1 : Visual question answering, grounding, and chain-of-thought reasoning with Argus. “ctx-token” is short for context token. 1 Introduction Recent breakthroughs in the training of multimodal large language models (MLLMs) Team 2023c ; Team 2024a ; Team 2023a ; Liu et al. 2024a ; Shi et al. 2025 ; Tong et al. 2024a ; Li et al. 2024b ; xAI Team 2024 ; Li et al. 2024a ; Lin et al. 2024 have unlocked great advancements in visual-language fusion, allowing these models to extract meaningful content from complex images and perform sophisticated reasoning tasks. However, predominately driven by the success from stronger large language models (LLMs), existing MLLMs still underperform in many vision-centric scenarios Tong et al. 2024b ; Tong et al. 2024a ; Shi et al. 2025 , where accurate visual perception and understanding determine the success of the subsequent multimodal reasoning tasks ( e.g., spatial relationship between objects or properties of specific regions-of-interests (RoIs)). To address these challenges, we revisit the design space of MLLMs from a vision-centric perspective , draw insights from cognitive visual intelligence, and propose a new visual attention grounding mechanism for multimodal reasoning tasks, as shown in Figure 1 . * * footnotetext: Work done during an internship at NVIDIA. † † footnotetext: Equal advising. Seminal studies in cognitive science Corbetta and Shulman 2002 ; James 1984 have recognized two distinct types of visual attention mechanism, stimulus-driven visual attention and goal-directed visual attention, which are also referred to as involuntary attention and voluntary attention Mountcastle 1978 ; Kaplan and Berman 2010 ; James 1984 ; Morecraft et al. 1993 , respectively. Stimulus-driven visual attention is an automatic bottom-up capture of attention driven by salient objects or textures in the visual stimulus. On the other hand, goal-directed attention is a top-down conscious selection of attention, driven by goals and intentions. Surprisingly, an interesting analogy of the two mechanisms of visual attention is presented in the design space of the MLLMs, where (1) the image tokenization with pre-trained visual foundation models Radford et al. 2021 ; Kirillov et al. 2023b ; Oquab et al. 2024 represents the stimulus-driven attention, and (2) the language-conditioned image feature engagement happening within the LLM’s transformer layers represents the goal-driven attention. The illustration of the visual attention activation strengths in Figure 2 clearly demonstrates different focus areas of two attention modules in MLLMs. Although several existing methods Zhang et al. 2024b ; Tong et al. 2024a ; Shi et al. 2025 have studied and highlighted the importance of unconditioned image tokenization to modern MLLMs’ reasoning capacity through knowledge distillation and mixture-of-vision-experts (MoVEs), the effect of explicit language-guided visual engagement is underexplored in the research community. This raises two natural questions: 1) What is the best way to introduce a language-directed visual attention mechanism into the design of MLLMs? 2) In addition to perception tasks, can a more explicit visual engagement benefit multimodal reasoning tasks? To answer these questions, we explore and propose a grounding-driven visual attention re-engagment module in the multimodal causal prediction process. Unlike most existing MLLMs relying on an implicit self-attention mechanism to model language-directed visual token attendance Liu et al. 2024a ; Tong et al. 2024a , we pivot to an explicit top-down visual search to locate the image RoI most relevant to the text prompt, and then guide the model to focus on the searched regions for subsequent reasoning and answer generation. Recent work has shown that an object-centric representation benefits the vision-language alignment process and subsequent perception tasks Chen et al. 2024a ; You et al. 2023 . Hence, we utilize text-to-box object-centric grounding as the intermediate reasoning stage, where the predicted bounding boxes serve as simple but effective visual chain-of-thought (CoT) signals to help improve the quality of the final reasoning step. The proposed method, Argus , is benchmarked across a diverse set of evaluation datasets, excelling not only in multimodal visual reasoning tasks Yue et al. 2024 ; Tong et al. 2024b ; Wu and Xie 2024 ; Tong et al. 2024a ; xAI Team 2024 ; Masry et al. 2022 ; Singh et al. 2019 ; Mathew et al. 2021 ; Hudson and Manning 2019 , but also in object-centric visual grounding tasks Kazemzadeh et al. 2014 ; Yu et al. 2016 . We also systematically analyze different designs of the visual attention re-engagement mechanism, and its collaboration with involuntary attention in MLLMs. We hope that our study paves the way towards stronger multimodal intelligence by emphasizing more vision-centric and perception-driven reasoning mechanisms. Figure 2 : Illustration of two visual attention mechanisms. Involuntary Attention (Left) : stimulus-driven; unconditioned feature extraction; salient objects. Direct Attention (Right) : Goal-driven; language-guided region-of-interest (RoI) feature extraction. 2 Related Work Visual Reasoning with MLLMs. The emergence of multimodal large language models (MLLMs) has revolutionized visual reasoning capabilities, allowing sophisticated question-answering and complex visual understanding tasks. Visual instruction tuning Liu et al. 2023c pioneers this advancement by establishing a foundation for tuning language models to handle multimodal tasks effectively. Following this, several improvements and architectural innovations have emerged to enhance zero-shot generalization abilities of MLLMs, including better visual-linguistic alignment Lin et al. 2024 ; Chen et al. 2024c ; Gao et al. 2024 ; Bai et al. 2023 , high-resolution visual input Liu et al. 2024a ; Li et al. 2024a ; Wang et al. 2024 , and dataset curation pipelines Chen et al. 2024d ; Tong et al. 2024a ; Wang et al. 2024 . This progress extends to proprietary models like GPT-4V Team 2023c , Claude 3 Team 2024a , and Gemini Team 2023a ; Team 2024d , which have showcased remarkable general-purpose applicability. Several recent studies have shifted focus towards exploring visual reasoning from a vision-centric perspective. Cambrian-1 Tong et al. 2024a conducts comprehensive investigations into various visual encoder architectures and introduces a specialized benchmark CV-Bench Tong et al. 2024a for assessing vision-centric reasoning capabilities. Eagle Shi et al. 2025 further advances this direction by introducing the mixture-of-vision-experts mechanism, demonstrating the potential of specialized visual processing highways in MLLM architectures. However, despite these advancements, current approaches lack conscious control over visual attention mechanisms and do not incorporate explicit goal-driven strategies for visual token extraction, which motivates the introduction of our method. Visual Perception with MLLMs. Visual perception has consistently been a critical