LogiDebrief: A Signal-Temporal Logic based Automated Debriefing Approach with Large Language Models Integration

Zirong Chen, Ziyan An, Jennifer Reynolds, Kristin Mullen, Stephen Martini, Meiyi Ma · 2025

Read LogiDebrief: A Signal-Temporal Logic based Automated Debriefing Approach with Large Language Models Integration by Zirong Chen, Ziyan An, Jennifer Reynolds, Kristin Mullen, Stephen Martini, Meiyi Ma (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03985.txt.

Introduction

Zirong Chen and co-authors present in 2025 an automated system for grading emergency call takers after 911 calls. Human review of call takers is slow and covers few calls, yet steady quality matters for public safety. Their system joins formal logic rules for call handling with large language models that read the calls.

The system works in three steps: it reads the context, checks the call against the rules, and rolls results into quality reports. In Nashville, it helped review 1,701 real calls and saved many hours of staff time. Tests also cover how well it grades calls. Readers see a working path to full, fast review of emergency calls.

Catalog record

CREATORZirong Chen, Ziyan An, Jennifer Reynolds, Kristin Mullen, Stephen Martini, Meiyi Ma
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-03985

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LogiDebrief: A Signal-Temporal Logic based Automated Debriefing Approach with Large Language Models Integration Zirong Chen, Ziyan An, Jennifer Reynolds, Kristin Mullen, Stephen Martini, Meiyi Ma Abstract Emergency response services are critical to public safety, with 9-1-1 call-takers playing a key role in ensuring timely and effective emergency operations. To ensure call-taking performance consistency, quality assurance is implemented to evaluate and refine call-takers’ skillsets. However, traditional human-led evaluations struggle with high call volumes, leading to low coverage and delayed assessments. We introduce LogiDebrief 1 1 1 More details: https://meiyima.github.io/angie.html , an AI-driven framework that automates traditional 9-1-1 call debriefing by integrating Signal-Temporal Logic (STL) with Large Language Models (LLMs) for fully-covered rigorous performance evaluation. LogiDebrief formalizes call-taking requirements as logical specifications, enabling systematic assessment of 9-1-1 calls against procedural guidelines. It employs a three-step verification process: (1) contextual understanding to identify responder types, incident classifications, and critical conditions; (2) STL-based runtime checking with LLM integration to ensure compliance; and (3) automated aggregation of results into quality assurance reports. Beyond its technical contributions, LogiDebrief has demonstrated real-world impact. Successfully deployed at Metro Nashville Department of Emergency Communications , it has assisted in debriefing 1,701 real-world calls, saving 311.85 hours of active engagement. Empirical evaluation with real-world data confirms its accuracy, while a case study and extensive user study highlight its effectiveness in enhancing call-taking performance. 1 Introduction Emergency response services are vital to public safety, with 9-1-1 call-takers as the first point of contact in crises, directly influencing response times and life-saving outcomes. Given their critical role, maintaining high performance is essential. To ensure consistency, emergency communication centers implement quality assurance programs that evaluate call-taker performance, enforce protocols, and provide actionable feedback. These programs use call reviews, guidecard cross-referencing, and protocol verification to enhance efficiency and emergency response effectiveness. Despite their critical role, quality assurance programs across the U.S. face challenges in providing timely feedback due to high call volumes and limited resources ( International, 2020 ; IAED and NASNA, 2023 ) . For example, during peak periods in 2024, the NYC Fire Department handled up to 6,500 emergency calls daily ( NY, 2025 ) , straining quality assurance personnel. As urban populations grow and emergency call volumes rise, these challenges intensify ( Ma et al., 2019 ) . Funding shortages and staffing constraints further hinder timely quality reviews ( Afonso, 2021 ) . Timely feedback is essential for effective quality assurance ( Adarkwah, 2021 ; Ahea et al., 2016 ) . Delays reduce relevance, making it harder for call-takers to recall key details, address performance gaps, and reinforce best practices. Without prompt debriefing, quality assurance programs risk becoming bottlenecks, delaying critical insights needed to improve call-taker training and emergency response. If unaddressed, these challenges may compromise 9-1-1 call centers’ ability to maintain high-performance standards, ultimately affecting response times and life-saving interventions. Given these demands, an automated system is urgently needed for effective emergency call debriefing. While LLMs have advanced natural language processing ( Wei et al., 2023 ; Rouzegar and Makrehchi, 2024 ) , their application in this domain presents significant challenges. Our preliminary trials identify three key challenges : (1) Step-by-step reasoning is crucial for evaluating call-taker performance, as emergency calls require strict procedural adherence. LLMs must not only understand context but also apply structured reasoning. While In-Context Learning (ICL) techniques, such as Chain-of-Thought (CoT) prompting ( Wei et al., 2022 ) , improve reasoning, studies show that LLMs still struggle with complex, high-stakes decision-making ( Miao et al., 2023 ; Huang et al., 2023 ; Kambhampati, 2024 ) . Even advanced automatic reasoning methods ( Zelikman et al., 2022 ) exhibit weaknesses in scenarios requiring rigorous procedural verification ( Wu et al., 2024 ; McCoy et al., 2024 ) . (2) Cross-document retrieval and reference remains another challenge. Call-takers rely on many complicated procedural documents, including guidecards, policies, and emergency protocols. While Retrieval-Augmented Generation (RAG) ( Lewis et al., 2020 ) assists by fetching external documents, its accuracy is inconsistent, often retrieving outdated or irrelevant information and struggling with multi-document synthesis ( Shi et al., 2023 ; Shuster et al., 2022 ) . Retrieval failures can result in missing critical protocols, significantly impacting evaluation. (3) The complex nature of emergency calls further complicates debriefing, as a single call may span multiple protocols (e.g., a motor vehicle accident may begin under police protocols but escalate to medical and fire due to injuries or hazards). Some calls exceed 20 minutes, making evaluation even more challenging. Combining ICL and RAG often results in excessively long prompts that exceed optimal context windows. Empirical studies ( Weng et al., 2024 ; Dong et al., 2024 ; An et al., 2024 ; Kuratov et al., 2024 ) show that longer prompts degrade performance, leading to incomplete reasoning, ignored context, and lower factual consistency, as illustrated in Figure 1 . In this paper, we introduce LogiDebrief , the first framework, to our knowledge, designed to automatically and effectively assist in 9-1-1 call-taking debriefing. LogiDebrief integrates logic-enhanced reasoning with LLMs’ language understanding, providing an effective approach to evaluating call-taker performance. Unlike traditional ICL methods that rely on lengthy prompts, LogiDebrief first collaborates with domain experts to decompose call-taking requirements into signal-temporal logic (STL) specifications ( Maler and Nickovic, 2004 ) . During runtime checking, LLMs function as independent evaluators within STL to verify compliance. Once verification is complete, LogiDebrief aggregates results, generates quality assurance forms, and delivers actionable feedback with tailored explanations. This automated, just-in-time debriefing process enhances call-taker training and improves emergency response effectiveness. Our technical innovations and contributions are: (1) We introduce LogiDebrief , a novel framework that automates 9-1-1 call-taking debriefing by integrating rigorous logic-based verification with LLM-powered analysis. (2) We decompose and formalize call-taking manuals into logic specifications through expert collaboration, ensuring standardized procedural verification, and improving consistency and reliability in 9-1-1 call debriefing. (3) We design an STL-integrated framework that seamlessly integrates LLMs as modular functions, enhancing procedural compliance while reducing the reliance on complex and lengthy prompts. (4) We empirically evaluate LogiDebrief’s performance through extensive experiments with real-world data, demonstrating its effectiveness in delivering accurate and reliable debriefing assessments. (5) We conduct a real-world case study and a user study (see user study in Appendix 8.1 ) under practical deployment. The findings confirm that LogiDebrief is an effective tool for improving call-taking performance and training in emergency response settings. Beyond its technical advancements, LogiDebrief delivers significant social impact : (1) Developed in collaboration with researchers and governmental agencies, LogiDebrief has been successfully deployed at Metro Nashville Department of Emergency Communications (MNDEC) . It is now integrated into training programs for both active call-takers and trainees. (2) To date, it has assisted in debriefing 1,701 real-world calls, saving an estimated 311.85 working hours. (3) It facilitates debriefing for more than 200 call scenarios, covering various responder departments, call types, and life-threatening situations. (4) LogiDebrief has the potential to scale nationwide, offering automated debriefing solutions to emergency communication centers, particularly those operating in resource-constrained environments. 2 Motivating Study Through discussions with MNDEC and a manual review of 1,244 past calls and debriefing results, we identified critical limitations in current 9-1-1 call center practices. Insufficient Call Review Coverage and Delayed Feedback Emergency dispatch centers

Read the complete text