VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment
Md. Mahfuzur Rahman, Kishor Datta Gupta, Marufa Kamal, Fahad Rahman, Sunzida Siddique, Ahmed Rafi Hasan, Mohd Ariful Haque, Roy George · 2025
Read VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment by Md. Mahfuzur Rahman, Kishor Datta Gupta, Marufa Kamal, Fahad Rahman, Sunzida Siddique, Ahmed Rafi Hasan, Mohd Ariful Haque, Roy George (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-21609.txt.
Introduction
General vision language models describe disaster images in generic words that responders cannot act on. Md. Mahfuzur Rahman and 7 co-authors build a knowledge enhanced captioning system in a 2025 paper. Their system, VLCE, targets post disaster satellite and drone imagery. External knowledge comes from ConceptNet and WordNet, two public semantic networks. A domain vocabulary of many terms is extracted from those graphs.
A standard model first captions the image, conditioned on object detections from YOLOv8. A second, knowledge enriched model then refines the caption. Two designs are tested for refinement: a CNN-LSTM and a hierarchical cross modal transformer. Benchmarks cover satellite and drone disaster data sets with damage classes. Semantic alignment and informativeness metrics score the results.
Catalog record
| CREATOR | Md. Mahfuzur Rahman, Kishor Datta Gupta, Marufa Kamal, Fahad Rahman, Sunzida Siddique, Ahmed Rafi Hasan, Mohd Ariful Haque, Roy George |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-21609
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VLCE: A Knowledge-Enhanced Framework for Image Description in Disaster Assessment Md. Mahfuzur Rahman, Kishor Datta Gupta, Marufa Kamal, Fahad Rahman, Sunzida Siddique, Ahmed Rafi Hasan, Mohd Ariful Haque, Roy George Abstract Immediate damage assessment is essential after natural catastrophes; yet, conventional hand evaluation techniques are sluggish and perilous. Although satellite and unmanned aerial vehicle (UAV) photos offer extensive perspectives of impacted regions, current computer vision methodologies generally yield just classification labels or segmentation masks, so constraining their capacity to deliver a thorough situational comprehension. We introduce the Vision Language Caption Enhancer (VLCE), a multimodal system designed to produce comprehensive, contextually-informed explanations of disaster imagery. VLCE employs a dual-architecture approach: a CNN-LSTM model with a ResNet50 backbone pretrained on EuroSat satellite imagery for the xBD dataset, and a Vision Transformer (ViT) model pretrained on UAV pictures for the RescueNet dataset. Both systems utilize external semantic knowledge from ConceptNet and WordNet to expand vocabulary coverage and improve description accuracy. We assess VLCE in comparison to leading vision-language models (LLaVA and QwenVL) utilizing CLIPScore for semantic alignment and InfoMetIC for caption informativeness. Experimental findings indicate that VLCE markedly surpasses baseline models, attaining a maximum of 95.33% on InfoMetIC while preserving competitive semantic alignment. Our dual-architecture system demonstrates significant potential for improving disaster damage assessment by automating the production of actionable, information-dense descriptions from satellite and drone photos. Keywords: Disaster Image Captioning; Vision-Language Models; Knowledge Graph Augmentation; Multimodal Deep Learning; Remote Sensing Analysis; Emergency Response Systems; Semantic Knowledge Integration 1 Introduction Global populations are particularly vulnerable to natural catastrophes, since floods, earthquakes, hurricanes, and wildfires cause extensive harm to ecosystems, infrastructure, and human lives all across the world. In these circumstances, quick damage assessment is essential for directing long-term restoration activities, assisting with search and rescue missions, and assigning emergency resources. However, in the immediate aftermath of disasters, typical manual assessment procedures are sometimes impracticable, slow, and dangerous. Recent advancements in computer vision and remote sensing technology present novel prospects for disaster damage evaluation. Satellite and unmanned aerial vehicle (UAV) imagery offer high-resolution perspectives of impacted regions, whereas deep learning frameworks have demonstrated significant efficacy in identifying and segmenting damage from aerial photographs [ 11 ] . Nonetheless, these techniques generally yield merely class labels or segmentation masks, constraining their capacity to deliver the detailed descriptive information essential for comprehending intricate disaster scenarios. To alleviate these challenges, image captioning has arisen as an effective method for visual interpretation and contextual comprehension. Image captioning is essential for assessing visual input and producing contextual descriptions that offer semantic information beyond rapid observation. Although photographs proficiently convey visual details and spatial relationships, they may not clearly express temporal context, causal links, or domain-specific interpretations. Image captioning augments visual information by transforming observations into descriptive, comprehensible text, so improving accessibility, interpretability, and actionability for decision-makers. Example (Disaster Scenario): Figure 1: RescueNet disaster image of post-hurricane devastation contrasting basic visual description with detailed automatic caption generation. Without caption: “A satellite image depicts a community from an aerial perspective, revealing multiple dwellings and streets” With caption: “The image shows the aftermath of Hurricane Michael, which inflicted major damage to infrastructure and the environment. Roads and streets are cluttered with debris from destroyed structures, such as fallen trees and scattered items. Trees in the region have suffered varied degrees of damage, with some seeming damaged or totally fallen. Buildings show different degrees of destruction, ranging from comparatively intact ones to those with obvious traces of catastrophic damage. The entire scene depicts the ongoing recovery and rebuilding activities in the aftermath of the catastrophe.” This contrast demonstrates how automated captioning goes beyond simple visual description to offer contextual knowledge. Responders may now make better decisions because the improved caption makes it apparent what kind of disaster it is, what infrastructure is impacted, and how far along the recovery is. As a result, image captioning goes beyond simple visual description to offer situational awareness and useful intelligence that is essential for applications involving accessibility, surveillance, and disaster response. Prior investigations into disaster image analysis have predominantly concentrated on remote sensing methodologies for damage assessment and disaster categorization. Many research have employed satellite image segmentation and change detection techniques [ 10 ] , although these methods fail to produce descriptive text summaries necessary for thorough situational analysis. Object detection in disaster imagery presents challenges across diverse settings [ 7 ] , and the precise identification of disaster-affected regions from aerial photography offers substantial difficulty for machine learning models [ 9 ] . Multimodal approaches signify sophisticated methodologies for disaster assessment [ 12 ] , with models such as LLaVA [ 13 ] showcasing notable progress in vision-language integration. Vision-Language Models (VLMs) have recently revolutionized automated image captioning capabilities [ 5 ] . These models show significant generalization across various domains; however, they lack proficiency in specialized tasks, underscoring the disparity between generalist capabilities and domain-specific expertise [ 6 ] . In critical scenarios like disasters, precise, detailed, and contextually relevant captions are vital for effective situational comprehension [ 14 ] . Image captioning in disaster contexts poses distinct challenges stemming from intricate environments, ambiguous object boundaries, and specialized content requirements. Although general-purpose VLMs may exhibit enhanced overall performance, they frequently lack the specialized knowledge required for effective interpretation of disaster scenes. To overcome these limitations, we propose VLCE (Vision Language Caption Enhancer), a specialized image captioning framework that enhances caption generation by integrating knowledge for the analysis of disaster scenarios. This approach combines visual features with external semantic knowledge from ConceptNet and WordNet to improve semantic understanding and produce descriptive, contextually relevant captions. Our model utilizes cross-modal attention mechanisms within tailored architectures to connect raw visual signals with significant, domain-specific textual descriptions. This framework specifically targets the generation of descriptions for satellite and drone imagery in disaster-affected regions, thereby addressing a notable deficiency in automated disaster assessment capabilities. The key contributions of this research include: • Creation of an innovative dual-architecture framework employing cross-modal attention processes for catastrophe picture captioning, featuring dedicated models for satellite and UAV imagery. • Incorporation of external knowledge graphs to augment vocabulary breadth and enhance semantic precision in disaster-related narratives. • Comprehensive disaster semantic alignment and caption informativeness evaluation utilizing CLIPScore and InfoMetIC. Our VLCE framework produces concise and useful descriptions of disaster imagery, aiding disaster responders, researchers, environmental scientists, and GIS specialists in swift post-disaster evaluation and analysis. The technology will aid NGOs, local communities, and visually impaired individuals by offering accessible summaries that improve communication, decision-making, and situational awareness in emergency situations. 2 Related Works Researchers have explored numerous studies in this field. Among them, some few works closely related to our research objectives are discussed below: Author of Zhai et al. [ 33 ] proposes a YOLOv8 model for the detection of UAVs using SPD-Conv, and GAM attention with precision, recall, and mAP by 11.9%, 15.2%, and 9%. Similarly, the author of Wang et al. [ 34 ] highlights a UAV aerial image on YOLOv8 model includes with WIoU v3 for better localisation, BiFormer attention for better feature attention, and a FasterNet block with new detection with a 7.7% accuracy. Another author [ 35 ] proposes six tuned YOLOv8 models for specific ranges of object sizes