GETReason: Enhancing Image Context Extraction through Hierarchical Multi-Agent Reasoning
Shikhhar Siingh, Abhinav RawatChitta Baral, Vivek GuptaSchool of Computing and Augmented Intelligence, Arizona State University {ssiingh, arawat21, chitta, vgupt140}@asu.edu · 2025
Read GETReason: Enhancing Image Context Extraction through Hierarchical Multi-Agent Reasoning by Shikhhar Siingh, Abhinav RawatChitta Baral, Vivek GuptaSchool of Computing and Augmented Intelligence, Arizona State University {ssiingh, arawat21, chitta, vgupt140}@asu.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21863.txt.
Introduction
Shikhhar Siingh and three co-authors present GETReason, a framework that extracts deep context from news images. Photos of public events carry clues about place, time, and meaning that plain captions miss. Journalism and education both gain when those clues are read well.
The method sends the image through layered agents that reason step by step about event, time, and location. The team also built a new scoring metric that weights reasoning quality and alignment across those three facets. Tests show the layered approach links images to their broader event context with real insight. The work moves image understanding beyond surface description toward grounded interpretation.
Catalog record
| CREATOR | Shikhhar Siingh, Abhinav RawatChitta Baral, Vivek GuptaSchool of Computing and Augmented Intelligence, Arizona State University {ssiingh, arawat21, chitta, vgupt140}@asu.edu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21863
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
GETReason: Enhancing Image Context Extraction through Hierarchical Multi-Agent Reasoning Shikhhar Siingh, Abhinav RawatChitta Baral, Vivek GuptaSchool of Computing and Augmented Intelligence, Arizona State University {ssiingh, arawat21, chitta, vgupt140}@asu.edu Abstract Publicly significant images from events carry valuable contextual information with applications in domains such as journalism and education. However, existing methodologies often struggle to accurately extract this contextual relevance from images. To address this challenge, we introduce GETReason ( G eospatial E vent T emporal Reason ing), a framework designed to go beyond surface-level image descriptions and infer deeper contextual meaning. We hypothesize that extracting global event, temporal, and geospatial information from an image enables a more accurate understanding of its contextual significance. We also introduce a new metric GREAT ( G eospatial, R easoning and E vent A ccuracy with T emporal alignment) for a reasoning capturing evaluation. Our layered multi-agentic approach, evaluated using a reasoning-weighted metric, demonstrates that meaningful information can be inferred from images, allowing them to be effectively linked to their corresponding events and broader contextual background. * * footnotetext: These authors contributed equally to this work. † † footnotetext: This corresponding author supervised the research. 1 Introduction Public event images capture moments of societal and historical importance—presidential inaugurations, mass protests, international summits—serving not only as visual records but as rich contextual artifacts. Understanding such images requires more than surface-level description; it demands inference about geopolitical, temporal, and event-specific factors often implicit in the scene. Despite their significance for journalism, archival analysis, and public discourse, event-centric image understanding remains underexplored. location: Indonesia ; time: October 20, 2019 event: Indonesian President Joko Widodo’s 2 {}_{\text{nd}} term Inauguration. Figure 1: Example from TARA Fu et al. 2022 dataset. Given an image, extract location and time. We add the inferred event. Most existing vision-language methods fall short in this setting. Traditional captioning models and encoder-decoder architectures describe visible content—objects, people, and actions—but fail to infer deeper meaning. Even advanced visual-language models Li et al. 2022 ; Hu et al. 2023 tend to narrate what is seen, missing why it matters. For instance, given an image of President Joko Widodo’s inauguration (Figure 1 ), these models might mention key figures but omit the event’s political gravity and temporal context. However, these methods fall short of true event understanding, as they fail to reason about temporal, spatial, and sociopolitical factors—many of which are implicit rather than directly observable. Efforts to move beyond superficial cues include reasoning-based methods Zhang et al. 2024 ; Song et al. 2024 , which can infer coarse event types like “ceremony,” and retrieval-augmented generation (RAG) Lewis et al. 2020 , which brings in external knowledge. Yet both have limitations: the former lacks specificity, and the latter is vulnerable to hallucinations and misinformation due to noisy sources Zeng et al. 2025 ; Deng et al. 2024 . These approaches often miss the nuanced, high-precision understanding required for real-world applications. Therefore, a robust solution is needed to automatically generate richer and more accurate narratives of public events, enhancing both human comprehension and downstream machine applications. Recent advances in multi-agent frameworks Dinh and Chan 2025 ; Ghafarollahi and Buehler 2024 ; Wang et al. 2025 ; Ng et al. 2024 demonstrate that dividing reasoning tasks among specialized agents can significantly improve contextual understanding across domains. However, these methods have primarily focused on textual or code-based settings. This raises an important question: Can a structured multi-agent framework effectively support contextual reasoning over complex visual data, such as public event images? To answer this question, we introduce GETReason , a structured multi-agent framework for extracting rich, contextual narratives from public event images. GETReason decomposes the task into three core sub-problems, each handled by a dedicated agent: (1) geospatial inference (e.g., “Indonesia”), (2) temporal inference (e.g., “October 20, 2019”), and (3) event-specific interpretation (e.g., “inauguration of President Joko Widodo’s second term amid political unrest”). Building on the divide-and-conquer principles of recent multi-agent systems Ng et al. 2024 ; Dinh and Chan 2025 , GETReason enables collaborative cross-agent validation to enhance factual accuracy, reduce hallucinations, and improve generalization across diverse event types. We evaluate GETReason on a large-scale dataset of public event images and demonstrate substantial gains over existing captioning and reasoning baselines. Our results highlight the importance of structured, role-specialized reasoning in achieving precise and reliable event understanding—unlocking new capabilities for downstream applications in news generation, historical archiving, and multimodal analysis. In summary, our key contributions are as follows: • Structured Multi-Agent Extraction Framework: We introduce a framework that systematically extracts event-related details while mitigating hallucinations Xu et al. 2018 , ensuring that VLMs focus on accurate and contextually relevant information rather than generating misleading inferences. • Evaluation Metric for Reasoning Quality: We propose a novel yet straightforward evaluation metric that inherently incorporates the reasoning process, allowing for a more meaningful assessment of the quality of extracted information. • Augmentation of Event-Centric Datasets: We address limitations in existing datasets by augmenting them with the attributes required for effective event extraction, ensuring better alignment with the problem domain. • Systematic Evaluation of Reasoning Strategies: Using carefully selected subsets of the augmented datasets, we rigorously evaluate the reasoning capabilities of VLMs, analyzing the specific strategies they employ to extract event-related details. The code and dataset are available at https://coral-lab-asu.github.io/getreason/ 2 Task Description We define the task of event-centric image understanding as generating a structured triplet from a single image depicting a public event. Given an image I , the model must infer: (1) a location L (e.g., city or country), (2) a temporal reference T (e.g., specific date or period), and (3) an event description E that captures the scene’s sociopolitical significance. Unlike standard captioning, which describes visible entities or actions, this task requires reasoning about partially implicit geopolitical, temporal, and narrative cues. In Figure 1 , for instance, the model links the image of President Joko Widodo to an oath-taking ceremony, the green-bound 1945 Constitution, and the presence of key political figures to derive the structured output: Expected Output : L : “Jakarta, Indonesia”; T : “October 20, 2019”; E : “Inauguration of President Joko Widodo’s second term amid protests.” Each component involves distinct reasoning challenges: L depends on geographic signals (e.g., flags, landmarks); T draws on visible dates, seasonal features, or known timelines; and E requires synthesizing the visual context with latent knowledge of political and historical events. This formulation enables structured, interpretable outputs for applications such as news analysis, archival indexing, and societal trend monitoring, while posing challenges in multimodal inference and hallucination control. 3 Dataset Description We use two dataset for our evaluation: WikiTilo Zhang et al. 2024 and TARA Fu et al. 2022 dataset. These dataset have images with temporal and geospatial information. WikiTiLo Dataset. The WikiTiLo dataset consists of 6,296 images annotated with specific spatio-temporal information, covering over 30 countries and spanning the years from 1826 to 2021. This dataset provides a diverse range of images with annotations detailing the time and country where each image was captured. TARA Dataset. The TARA dataset comprises of approximately 16,000 images associated with news articles, along with their corresponding time and location information, automatically extracted from the New York Times (NYT) † † † https://developer.nytimes.com/docs/archive-product/1/overview . For our evaluation, we utilized the train