Towards Automated Situation Awareness: A RAG-Based Framework for Peacebuilding Reports
Poli A. Nemkova, Suleyman O. Polat, Rafid I. Jahan, Sagnik Ray Choudhury, Sun-joo Lee, Shouryadipta Sarkar, Mark V. Albert · 2025
Read Towards Automated Situation Awareness: A RAG-Based Framework for Peacebuilding Reports by Poli A. Nemkova, Suleyman O. Polat, Rafid I. Jahan, Sagnik Ray Choudhury, Sun-joo Lee, Shouryadipta Sarkar, Mark V. Albert (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-10586.txt.
Introduction
Peacebuilding teams need fast, trustworthy reads on fast-moving crises, and a 2025 paper by Poli Nemkova and co-authors builds a system for that need. Human analysts face vast, mixed sources, from news to conflict logs to economic data, and manual work causes delays. Their system fetches live sources and drafts situation reports on demand.
Each query builds its own small knowledge base, so reports stay current and on topic. Quality is checked at three levels: auto scores for clarity and fact fit, expert review, and user feedback. The frame targets early warning and aid planning, where hours matter. Shared checks aim to keep speed from trading away trust.
Catalog record
| CREATOR | Poli A. Nemkova, Suleyman O. Polat, Rafid I. Jahan, Sagnik Ray Choudhury, Sun-joo Lee, Shouryadipta Sarkar, Mark V. Albert |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-10586
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Automated Situation Awareness: A RAG-Based Framework for Peacebuilding Reports Poli A. Nemkova, Suleyman O. Polat, Rafid I. Jahan, Sagnik Ray Choudhury, Sun-joo Lee, Shouryadipta Sarkar, Mark V. Albert Abstract Timely and accurate situation awareness is vital for decision-making in humanitarian response, conflict monitoring, and early warning and early action. However, the manual analysis of vast and heterogeneous data sources often results in delays, limiting the effectiveness of interventions. This paper introduces a dynamic Retrieval-Augmented Generation (RAG) system that autonomously generates situation awareness reports by integrating real-time data from diverse sources, including news articles, conflict event databases, and economic indicators. Our system constructs query-specific knowledge bases on demand, ensuring timely, relevant, and accurate insights. To ensure the quality of generated reports, we propose a three-level evaluation framework that combines semantic similarity metrics, factual consistency checks, and expert feedback. The first level employs automated NLP metrics to assess coherence and factual accuracy. The second level involves human expert evaluation to verify the relevance and completeness of the reports. The third level utilizes LLM-as-a-Judge, where large language models provide an additional layer of assessment to ensure robustness. The system is tested across multiple real-world scenarios, demonstrating its effectiveness in producing coherent, insightful, and actionable reports. By automating report generation, our approach reduces the burden on human analysts and accelerates decision-making processes. To promote reproducibility and further research, we openly share our code and evaluation tools with the community via GitHub. 1 Introduction Timely situation awareness is essential for effective decision-making in peacekeeping operations, humanitarian response, and governmental interventions. Organizations such as the United Nations (UN), non-governmental organizations (NGOs), and policy-makers rely on accurate and up-to-date reports to allocate resources, anticipate crises, and mitigate conflict escalation. However, generating these reports manually is time-intensive, requiring extensive data collection, synthesis, and expert analysis. The complexity of integrating heterogeneous, real-time data from multiple sources further exacerbates this challenge, leading to delays that can hinder timely interventions. Advancements in large language models (LLMs) have demonstrated strong capabilities in text aggregation and summarization. However, standard LLM-based approaches face limitations such as hallucination, inconsistencies, and limited fact verification, making them unreliable for high-stakes decision-making. Retrieval-Augmented Generation (RAG) offers a promising alternative by integrating retrieval-based fact grounding with generative models. By dynamically retrieving and incorporating relevant, real-world data into the response generation process, RAG significantly improves factual consistency while reducing hallucinations. In this paper, we propose a multi-modal dynamic RAG framework designed for automated situation awareness reporting. Our system ingests real-time data from diverse sources, including news articles, conflict event databases, and economic indicators, constructing query-specific knowledge bases to ensure contextually relevant and temporally prompt insights. Reports generated through this system provide stakeholders with structured, evidence-backed summaries, reducing the time and effort required for manual analysis. To assess the reliability and effectiveness of our system, we introduce a three-level evaluation framework that includes: Automated NLP-Based Metrics – Assessing factual accuracy, completeness, and coherence using semantic similarity models, factual consistency checks, and bias detection tools. Human Expert Review – Evaluating relevance, completeness, and usability through domain experts from the UN. LLM-as-a-Judge Assessment – Benchmarking the model’s performance using LLM-based evaluation, comparing outputs with human judgments. We evaluate our approach across multiple real-world scenarios and demonstrate that our RAG system produces coherent, insightful, and actionable reports while reducing the burden on human analysts. By automating the early-stage report generation process, we accelerate decision-making while maintaining expert oversight. To encourage further research and reproducibility, we openly share our code, evaluation framework, and sample reports with the community. To the best of our knowledge, this study presents the first application of a Retrieval-Augmented Generation (RAG) framework integrated with large language models (LLMs) in the peacebuilding domain. Furthermore, it constitutes the first documented instance of AI-driven automation leveraging publicly available data to support decision-making processes for high-impact non-governmental organizations (NGOs), including the United Nations, a key partner in this project. As a technical contribution, we introduce a dynamic RAG implementation coupled with a three-level reference-free evaluation framework. Additionally, we developed a modified version of VERISCORE Song et al. 2024 , packaged as ragve , which enables verification that the RAG-generated outputs are grounded solely in the retrieved data—a valuable tool for researchers working on similar models. The ragve package, along with our full implementation, is publicly available on GitHub. 2 Literature Review Retrieval-Augmented Generation (RAG) represents an advanced methodology that integrates the generative power of large language models (LLMs) with the precision of information retrieval systems. By leveraging external knowledge sources, RAG frameworks enhance the contextual relevance and factual accuracy of generated content. This hybrid approach has demonstrated substantial benefits across diverse fields such as safety, finance, healthcare, and scientific research, significantly improving the quality and efficiency of automated reporting Gao et al. 2023 ; Fan et al. 2024 ; Arslan et al. 2024 . In safety-critical domains , RAG frameworks have been effectively customized to generate comprehensive reports based on session logs and incident descriptions. For instance, in the aviation sector, models like LLaMA combined with embedding techniques have automated safety report generation, achieving significant improvements in accuracy and compliance with documentation standards. Metrics such as Recall@5, GLEU, METEOR, and BERTScore highlight the superiority of RAG-based methods over traditional reporting systems Bernardi et al. 2024 ; Suresh et al. 2024 . In the financial sector , RAG has enhanced the analysis and interpretation of complex financial reports, particularly in question-answering tasks for private investors. Utilizing models like OpenAI’s ADA and GPT-4, these systems process half-yearly and quarterly reports with high contextual relevance and accuracy. Research underscores that well-structured financial documents further optimize RAG performance, especially in addressing qualitative queries Iaroshev et al. 2024 . Similarly, in healthcare , RAG has been employed to automate the generation of radiology reports through multimodal embeddings, facilitating the retrieval of relevant clinical information. Generative models, such as OpenAI’s GPT series, integrate user-specific clinical requirements to improve report quality while mitigating hallucination risks. These systems have achieved enhanced performance on clinical metrics like BERTScore and Semb score, showcasing the framework’s utility in high-stakes environments Ranjit et al. 2023 ; Markey et al. 2024 ; Wang et al. 2024 ; Assistant et al. 2024 ; Alam et al. 2024 . While RAG has proven successful across these domains, its application in peacebuilding remains an emerging area of exploration. Recent advances in Natural Language Processing (NLP) have demonstrated the potential of LLMs and RAG frameworks for conflict prediction and monitoring human rights violations. Studies have successfully applied NLP techniques to detect armed conflicts and human rights abuses using diverse data sources, including social media and news reports Trivedi et al. 2020 ; Alhelbawy et al. 2020 ; Mueller et al. 2024 ; Nemkova et al. 2023 . However, comprehensive, automated situation awareness reporting for conflict prevention, resolution, and post-conflict recovery is still underdeveloped. Our work seeks to bridge this gap by extending the proven capabilities of RAG frameworks to peacebuilding, offering a robust solution for humanitarian and governmental organizations. The evolution of multimodal RAG systems has further expanded report generation capabilities by incorporating diverse data types, including text, tables, and images. These systems enhance retrieval and content generation by leveraging interrelationships between different modalities, thereby improving the contextual richness and accuracy