Sim911: Towards Effective and Equitable 9-1-1 Dispatcher Training with an LLM-Enabled Simulation

Zirong Chen, Elizabeth Chason, Noah Mladenovski, Erin Wilson, Kristin Mullen, Stephen Martini, Meiyi Ma · 2024

Read Sim911: Towards Effective and Equitable 9-1-1 Dispatcher Training with an LLM-Enabled Simulation by Zirong Chen, Elizabeth Chason, Noah Mladenovski, Erin Wilson, Kristin Mullen, Stephen Martini, Meiyi Ma (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-16844.txt.

Introduction

Zirong Chen and six co-authors introduce Sim911, a simulation for training emergency dispatchers with large language models. Traditional training relies on experienced staff role-playing callers, which is slow, costly, and often overlooks underserved communities. A realistic simulator could widen access to practice.

Sim911 builds its scenarios from archived emergency call data so practice mirrors real incidents. Dynamic prompts and vector bases steer the model's behavior toward training goals. A validation loop filters weak responses and refines output quality over time. The authors present the system as the first of its kind for dispatcher training. Its stated aim is training that is both effective and equitable.

Catalog record

CREATORZirong Chen, Elizabeth Chason, Noah Mladenovski, Erin Wilson, Kristin Mullen, Stephen Martini, Meiyi Ma
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-16844

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sim911: Towards Effective and Equitable 9-1-1 Dispatcher Training with an LLM-Enabled Simulation Zirong Chen, Elizabeth Chason, Noah Mladenovski, Erin Wilson, Kristin Mullen, Stephen Martini, Meiyi Ma Abstract Emergency response services are vital for enhancing public safety by safeguarding the environment, property, and human lives. As frontline members of these services, 9-1-1 dispatchers have a direct impact on response times and the overall effectiveness of emergency operations. However, traditional dispatcher training methods, which rely on role-playing by experienced personnel, are labor-intensive, time-consuming, and often neglect the specific needs of underserved communities. To address these challenges, we introduce Sim911 1 1 1 More details: https://meiyima.github.io/angie.html , the first training simulation for 9-1-1 dispatchers powered by Large Language Models (LLMs). Sim911 enhances training through three key technical innovations: (1) knowledge construction, which utilizes archived 9-1-1 call data to generate simulations that closely mirror real-world scenarios; (2) context-aware controlled generation, which employs dynamic prompts and vector bases to ensure that LLM behavior aligns with training objectives; and (3) validation with looped correction, which filters out low-quality responses and refines the system performance. Beyond its technical advancements, Sim911 delivers significant social impacts. Successfully deployed in the Metro Nashville Department of Emergency Communications (MNDEC) , Sim911 has been integrated into multiple training sessions, saving time for dispatchers. By supporting a diverse range of incident types and caller tags, Sim911 provides more realistic and inclusive training experiences. In a conducted user study, 90.00% of participants found Sim911 to be as effective or even superior to traditional human-led training, making it a valuable tool for emergency communications centers nationwide, particularly those facing staffing challenges. 1 Introduction Emergency response services are essential for public safety, managing around 240 million 911 calls annually, based on year-round stats from New York City ( NYC-911 2022 ) . However, there is a critical staffing shortfall, with a third of centers reporting more vacancies in 2023 compared to 2019, resulting in approximately 25,000 unfilled positions nationwide. This staffing crisis increases the workload on current staff ( Chen et al. 2022 ; Chen et al. 2023 ) , leading to dispatcher burnout and impacting emergency service quality ( NICE 2023 ) . As urban areas in the US grow, the strain on emergency response systems intensifies. Rapid urbanization and population growth demand effective solutions to adapt to and manage these increasing pressures ( Ma et al. 2019 ) . Traditional training environments prepare trainees for real-world dispatcher roles by employing role-playing scenarios where experienced dispatchers coach trainees through simulated calls. The need for experienced dispatchers to participate in training diverts essential personnel from actual emergency duties, causing inconsistencies in training quality and reducing the availability of skilled staff, particularly in underserved areas ( Saxon et al. 2022 ; Afonso 2021 ) . However, traditional training methods, which rely on role-playing by experienced personnel, are labor-intensive, time-consuming, and frequently overlook the specific needs of underserved communities. In light of these demands, exploring innovative technological solutions is critical. Advancements in artificial intelligence, especially Large Language Models (LLMs), offer promising methods for enhancing training environments. Employing LLMs to simulate caller interactions can reduce reliance on human resources, improving training efficiency and consistency ( Naveed et al. 2023 ; Wang et al. 2023 ; Carta et al. 2023 ) . However, directly applying plain LLM agents is not ideal. In our preliminary investigations, we identified the following challenges : (1) Achieving consistent realistic simulations is difficult without detailed factual databases, despite meticulous prompt engineering across different LLMs. This lack of realism results in simulations that do not fit the local context, making the training less effective and potentially confusing for trainees. (2) LLMs excel in generating coherent content but tend to fabricate details, undermining authenticity . Simulations with fabricated geographic information lead dispatchers to make decisions based on incorrect data, compromising emergency response effectiveness. (3) The needs of vulnerable populations in metropolitan areas are often understudied during conventional training, leaving practitioners unprepared. This lack of inclusiveness results in biased training, inadequately preparing dispatchers for handling calls from vulnerable groups, and leading to disparities in emergency response. (4) The inherently complex nature of 9-1-1 calls presents significant challenges, even for human trainers. Human-led training may also fail to capture these complexities, as discussed in detail in Section 2 . In this paper, we introduce Sim911, the first system that leverages LLMs to simulate realistic 9-1-1 calls, specifically designed to enhance dispatcher training. Sim911 focuses on creating effective and equitable simulation experiences tailored to the local metro area. Sim911 comprises three key components: knowledge construction , context-aware controlled generation , and validation with looped feedback . Knowledge construction organizes real-world information into retrieval knowledge bases, while context-aware controlled generation fine-tunes the LLM’s behavior through human-designed instructions. Validation with looped feedback ensures high-quality outputs by filtering out low-quality responses. We summarize our technical innovations and contributions as follows: (1) Innovative Knowledge Construction from 9-1-1 Calls : Sim911 organizes real-world call data into detailed knowledge bases, allowing for the generation of contextually accurate and realistic training simulations, supporting 57 different incident types. (2) Context-Aware Controlled Generation : Sim911 strategically and dynamically uses advanced techniques, such as Chain-of-Thought (CoT) and Retrieval-Augmented Generation (RAG), to tailor LLM behavior during training sessions. (3) Validation with Looped Correction : Sim911 includes a unique validation process that filters out low-quality responses, ensuring high-quality and scenario-appropriate outputs. (4) Focus on Social Equity : Sim911 emphasizes training that addresses the needs of underserved and vulnerable communities, incorporating relevant data to better prepare dispatchers for diverse real-world scenarios. (5) Real-World Deployment and Evaluation : Successfully deployed in DEC, Sim911 has proven an effective tool in enhancing 9-1-1 dispatcher training from experimental results on real-world data. Beyond technical advancements, Sim911 delivers significant social impacts : (1) Sim911 has been successfully deployed in DEC’s training programs, seamlessly integrated into 4 training classes across different service sites. (2) To the date of this paper, Sim911’s system logs reveal a total active simulation time of 26.55 hours, effectively saving this time for MNDEC dispatchers. (3) Sim911 supports 57 different real-world incident types and covers 14 caller tags, such as “unhoused” and “non-English speaking,” to enrich caller profiles. (4) In a user study conducted with DEC, 90.00% of participants found Sim911 to be as effective or even superior to traditional human-led training. Additionally, Sim911 received an average helpfulness score of 4.89 for its assistance in call-taking training. (5) Sim911 has the potential to assist emergency communications centers across the U.S. with limited staffing by enabling trainees to engage individually with the training program. 2 Motivating Study We analyzed 11,841 real-world phone call recordings (from Nov. 2022 to May 2024) and manually reviewed 33 conventional training pieces (see examples in the Appendix, leading to the following observations. Traditional training is laborious and time-consuming . In traditional training setups, each trainee engages in call simulations, assuming three roles: the call-taker, the caller, and the instructor. The trainee, as the call-taker, manages calls directed by the instructor and an experienced dispatcher. Each trainee typically participates in 60 independent simulated calls, with each call requiring the involvement of three participants. Based on past recordings, the average call duration is 3.5 minutes; with an average of 12 trainees per session,

Read the complete text