DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenes

Aisha Al-Mohannadi, Ayisha Firoz, Yin Yang Muhammad Imran Ferda Ofli · 2026

Read DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenes by Aisha Al-Mohannadi, Ayisha Firoz, Yin Yang Muhammad Imran Ferda Ofli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-13839.txt.

Introduction

Aisha Al-Mohannadi and co-authors build a test set for AI questions on disaster photos in 2026. Social media images give fast, low-latency clues during natural and human-made disasters. Visual question answering, or VQA, lets users ask a model about an image. Strong results on plain photos do not prove skill in disaster scenes.

Their benchmark, DisasterVQA, targets damage assessment and response needs. Disaster reasoning is complex and safety-critical, unlike most VQA tasks. A firm test set shows where models fail when lives are at stake. Responders could then trust, or avoid, model help with clear eyes.

Catalog record

CREATORAisha Al-Mohannadi, Ayisha Firoz, Yin Yang Muhammad Imran Ferda Ofli
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-13839

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenes Aisha Al-Mohannadi, Ayisha Firoz, Yin Yang Muhammad Imran Ferda Ofli Abstract Social media imagery provides a low-latency source of situational information during natural and human-induced disasters, enabling rapid damage assessment and response. While Visual Question Answering (VQA) has shown strong performance in general-purpose domains, its suitability for the complex and safety-critical reasoning required in disaster response remains unclear. We introduce DisasterVQA, a benchmark dataset designed for perception and reasoning in crisis contexts. DisasterVQA consists of 1,395 real-world images and 4,405 expert-curated question–answer pairs spanning diverse events such as floods, wildfires, and earthquakes. Grounded in humanitarian frameworks including FEMA ESF and OCHA MIRA, the dataset includes binary, multiple-choice, and open-ended questions covering situational awareness and operational decision-making tasks. We benchmark seven state-of-the-art vision–language models and find performance variability across question types, disaster categories, regions, and humanitarian tasks. Although models achieve high accuracy on binary questions, they struggle with fine-grained quantitative reasoning, object counting, and context-sensitive interpretation, particularly for underrepresented disaster scenarios. DisasterVQA provides a challenging and practical benchmark to guide the development of more robust and operationally meaningful vision–language models for disaster response. The dataset is publicly available at https://zenodo.org/records/18267770 . Introduction Natural disasters such as floods, earthquakes, wildfires, and hurricanes cause widespread devastation, threatening lives, infrastructure, and economies. In their immediate aftermath, first responders and humanitarian agencies require timely and actionable intelligence to support response and recovery efforts. Rapidly understanding visual conditions on the ground is therefore critical for effective humanitarian operations. Social media plays an increasingly important role in modern disaster management by providing low-latency access to visual information from affected areas, provided that damage scenes can be accurately interpreted and systematically analyzed ( Imran et al. 2020 ; Agarwal et al. 2020 ) . To this end, Vision-Language Models (VLMs), which integrate visual and textual information, offer promising capabilities for automating the analysis of complex scenes. Existing Visual Question Answering (VQA) benchmark datasets have enabled significant progress in understanding the everyday scenes ( Goyal et al. 2017 ; Hudson and Manning 2019 ; Al-Tahan et al. 2024 ; Yu et al. 2024 ) . However, these datasets primarily focus on common objects, people, and daily activities, and do not capture the unique challenges posed by disaster imagery. Some Earth observation-oriented VQA benchmarks address disaster-related scenarios using overhead satellite imagery, but they typically focus on narrow, task-specific objectives such as disaster prediction or building damage assessment, often framed within broader general-purpose evaluation settings ( Danish et al. 2025 ; Soni et al. 2025 ; Wang et al. 2025 ) . In contrast, there is currently no VQA dataset that is (i) specifically curated for disaster scenes depicted in natural, ground-level imagery and (ii) explicitly grounded in established humanitarian disaster response frameworks. Disaster environments differ fundamentally from everyday settings: they are chaotic, partially damaged, and visually complex, requiring models to reason about critical conditions such as accessibility, damage severity, and safety risks. Without dedicated benchmarks, current VLMs cannot be systematically evaluated or effectively advanced for these high-stakes humanitarian applications. To tackle this challenge, we introduce DisasterVQA, the first large-scale VQA dataset centered on real-world natural disaster imagery. DisasterVQA consists of around 4,405 carefully annotated image-question-answer triplets, covering a diverse range of disaster events. The questions are designed across three formats: binary, multiple-choice, and open-ended, to comprehensively assess VLMs’ reasoning capabilities under varied levels of difficulty and ambiguity. Importantly, the questions are grounded in established humanitarian response frameworks (e.g., FEMA ESF ( FEMA 2025 ) , OCHA MIRA ( UN OCHA 2025 ) , etc.) and cover tasks such as identifying built environment damage, determining hazard type and severity, evaluating accessibility, and assessing movement restrictions and controls, making the dataset highly relevant for real-world disaster response applications. Figure 1 illustrates sample image-question-answer triplets from the dataset. Figure 1: Sample image-question-answer triplets from the proposed DisasterVQA dataset Alongside the dataset, we provide a comprehensive evaluation framework to drive reproducible research in disaster‑focused VQA, benchmarking seven state‑of‑the‑art VLM architectures (Molmo-7B, LLama-3.2-11B, Pixtral-12B, Mistral-Small-3.1-24B, Qwen2.5-VL-32B, GPT-4o-mini, GPT-4.1-mini) across binary, multiple‑choice, and open‑ended question types and reporting baseline performances in terms of accuracy, precision, recall, and F_{1} -score. The evaluation reveals that while binary questions are handled with relatively high reliability especially by GPT-4.1-mini, multiple-choice and open-ended questions remain challenging, with performance varying across disaster types and geographical regions. Open-weight models, though competitive in some areas, often show greater inconsistency. These results underscore the importance of region- and task-specific benchmarking in VQA, and highlight the need for improved visual reasoning in zero-shot settings. By exposing these strengths and limitations, our benchmark offers actionable insights for advancing VLMs in high-stakes, real-world disaster scenarios. Related Work VQA for disaster response remains underexplored, with few datasets that capture the complexity of real world crisis scenes. To our knowledge, the only dedicated benchmark, FloodNet ( Rahnemoonfar et al. 2021 ) , provides around 3,200 UAV images and 11,000 question-answer pairs for flood detection, road passability, and water-body differentiation. However, FloodNet is restricted to aerial images of flood scenarios with mostly binary and counting questions, lacking ground-level complexity and is limited in event diversity. In the literature, advanced prompting techniques to FloodNet were proposed, e.g., in ( Sun, Wang, and Peng 2024 ; Karimi and Rahnemoonfar 2025 ) ; however, these solutions are largely constrained by the narrow question formation and single-event focus of FloodNet. Figure 2: Overview of the four stages followed for the dataset curation and benchmarking Beyond flood-specific VQA, RSVQA ( Lobry et al. 2020 ) introduces VQA on satellite images for land cover classification and object counting. However, RSVQA depends upon aerial imagery as well as simple counting / classification questions, which do not support the detailed inferencing needed for disaster‑response reasoning. The work has been further improved in Prompt-RSVQA ( Chappuis et al. 2022 ) to introduce visual context prompts into language models, where the authors provide more hints such as object labels, and scene metadata to help improve model performance on RSVQA. However, its questions remain standard environmental monitoring rather than specialized, high-stakes questions that would be necessary for disaster response. A recent multimodal VQA dataset, TAMMI ( Boussaid et al. 2025 ) , includes RGB, multispectral, and SAR imagery for geospatial question answering, which could be valuable in disaster scenarios. However, TAMMI does not include questions assessing accessibility, damage severity, or safety in these contexts. Generic VQA benchmarks, such as VQA v2.0 ( Goyal et al. 2017 ) , GQA ( Hudson and Manning 2019 ) , and Visual7W ( Zhu et al. 2016 ) , mostly cover everyday scenes rather than disaster ones. The clean, object‑centric images in these datasets often fail to reflect the occlusions, debris, and visual chaos of disaster environments; meanwhile, despite the large number of images therein, these datasets do not address critical challenges in disaster response, such as poor visibility or partially damaged objects. In the disaster imagery domain beyond VQA, datasets such as xBD ( Gupta et al. 2019 ) compile massive pre- and post-event satellite imagery with around 850K building annotations and ordinal damage levels for segmentation and classification tasks. However, xBD lacks a question-answer interface needed for interactive reasoning about infrastructure saftey or rescue

Read the complete text