HomeEmergency - Using Audio to Find and Respond to Emergencies in the Home

James F. Mullen Jr, Dhruva Kumar, Xuewei Qi, Rajasimman Madhivanan, Arnie Sen, Dinesh Manocha, Richard Kim · 2025

Read HomeEmergency - Using Audio to Find and Respond to Emergencies in the Home by James F. Mullen Jr, Dhruva Kumar, Xuewei Qi, Rajasimman Madhivanan, Arnie Sen, Dinesh Manocha, Richard Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-01089.txt.

Introduction

James Mullen and co-authors teach home robots to find and judge emergencies by sound, in work from 2025. Accidental home deaths in the United States top 128,000 a year, many tied to fires, falls, and gas leaks. A robot that patrols the home could catch trouble early.

They built a test set of home scenes in a 3D simulator. Each run starts with a sound that may or may not signal danger. The robot must roam the rooms, mixing sound, sight, and memory to decide. Their system maps the home as a live graph and uses vision-language models to judge object risks, like flammability. A real-world demo shows the method working end to end.

Catalog record

CREATORJames F. Mullen Jr, Dhruva Kumar, Xuewei Qi, Rajasimman Madhivanan, Arnie Sen, Dinesh Manocha, Richard Kim
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-01089

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HomeEmergency - Using Audio to Find and Respond to Emergencies in the Home James F. Mullen Jr, Dhruva Kumar, Xuewei Qi, Rajasimman Madhivanan, Arnie Sen, Dinesh Manocha, Richard Kim Abstract In the United States alone accidental home deaths exceed 128,000 per year. Our work aims to enable home robots who respond to emergency scenarios in the home, preventing injuries and deaths. We introduce a new dataset of household emergencies based in the ThreeDWorld simulator. Each scenario in our dataset begins with an instantaneous or periodic sound which may or may not be an emergency. The agent must navigate the multi-room home scene using prior observations, alongside audio signals and images from the simulator, to determine if there is an emergency or not. In addition to our new dataset, we present a modular approach for localizing and identifying potential home emergencies. Underpinning our approach is a novel probabilistic dynamic scene graph (P-DSG), where our key insight is that graph nodes corresponding to agents can be represented with a probabilistic edge. This edge, when refined using Bayesian inference, enables efficient and effective localization of agents in the scene. We also utilize multi-modal vision-language models (VLMs) as a component in our approach, determining object traits (e.g. flammability) and identifying emergencies. We present a demonstration of our method completing a real-world version of our task on a consumer robot, showing the transferability of both our task and our method. Our dataset will be released to the public upon this papers publication. I Introduction Fig. 1: An overview of our method. Our agent hears a ”thud” and determines that a fall may have occurred. It then leverages the probabilistic edges of our probabilistic dynamic scene graph (P-DSG), representing a heatmap of agent activity, and audio direction to update our P-DSG and produce a hypothesized source location. In this example, the P-DSG shows a high probability of the user, and thus any fall, being in the office (the red box). The agent checks the office and detects a fallen person. It calls emergency services. Imagine you have an elderly parent at home and they take a hard fall. This situation is very serious and potentially deadly, with a quick emergency response essential to a positive outcome. In the United States in 2022 accidental home deaths exceeded 128,000, over three times the death rate of motor vehicles [ 1 ] . 34% of these deaths are due to falls directly with another 48% potentially leading to falls (poisoning or choking) [ 2 ] . Additionally, almost 7 million people in the United States were treated in emergency rooms for fall or trip related injuries in 2020 alone. Fires represent the next largest cause of death and damages, accounting for 10% of accidental home deaths and over $10 Billion in damages each year [ 3 ] . Now imagine a home robot that can monitor the home and respond to these types of emergencies quickly. Their ability to contact emergency services in a timely fashion would make a serious difference in outcome [ 4 ] . As a home robot cannot be present in multiple rooms at once, audio is a natural cue to detect possible emergencies in the home. The audio signal can be perceived the instant something occurs, enabling a speedy response. It also contains information about what may be occurring, like a ‘thud’ noise for a hard fall, or an alarm for a fire. The AudioGoal [ 5 ] task introduced Audio-Visual (AV) navigation with a goal of finding the source of a periodic (continuous) sound using egocentric images and captured binaural audio. This is similar to the ObjectGoal task but with audio instead of a target object. Subsequent works added shorter duration sounds, and semantic grounding of sounds [ 6 , 7 ] . For emergencies in a home environment, these tasks are inadequate. Additionally, no known tasks allow prior exploration of the scene, which a home robot will have. Main contributions. We introduce the HomeEmergency dataset to benchmark the abilities of embodied AI agents to identify emergencies in the scene. Figure 1 showcases our task. When initialized, an agent is spawned into the simulator and has the opportunity to explore the environment as a home robot would in the real world. At runtime, the robot is again placed into the simulator at a random point, but this time an audio signal is played in the environment representing a real emergency (a fallen person) or a false emergency (a fallen box or suitcase). The agent must efficiently find the source of the audio and determine if an emergency is occurring. The HomeEmergency task is challenging due to its aperiodic sounds, multiple rooms, and partial observability of the environment. Additionally, agents must not just find the source of the audio, but effectively determine whether an emergency has or has not occurred when presented with a true emergency or false emergency. We propose a modular method for this task consisting of perception, mapping, fusion, and emergency identification modules. These modules use the audio propagated through the scene to determine that an emergency may be occurring, before using the direction the audio came from and a probabilistic dynamic scene graph generated through previous exploration of the scene to determine the likeliest location of the audio source. The agent navigates to the audio source and uses the emergency identification module to determine if an emergency is occurring. Our main contributions are as follows: 1. The HomeEmergency dataset, aimed at enabling researchers to create embodied agents capable of responding to emergency situations in the home. We provide code for users to generate data samples. 2. A modular method for the HomeEmergencySim task, consisting of four components: perception, mapping, fusion, and emergency identification. 3. A novel scene graph structure which extends dynamic scene graphs to mobile objects or agents by using probabilistic edges, which represent the likelihood of the agent being in a given location. 4. A novel inference algorithm that uses Bayesian inference to update the probabilistic dynamic scene graph with new information and hypothesize the location of an agent for quicker response to emergencies. 5. We conduct a sim-to-real transfer of our method and demonstrate its performance. II Related Work II-A Audio-Visual Navigation Audio-Visual (AV) navigation was introduced in AudioGoal [ 5 ] where the agent has to navigate to the source of a periodic sound using egocentric images and captured binaural audio. Initial work focused on navigating to the sound source by inferring the source of the sound and moving towards it [ 8 , 9 ] . Chen et al. [ 10 ] builds upon these works by using reinforcement learning, setting waypoints, and mapping the audio to the scene. Subsequent research has brought AudioGoal closer to the real world. Chen et al. (SemanticAudioGoal) [ 6 ] limits sound length and localizes sounds to an object that makes semantic sense. Gan et al. [ 7 ] (FFO) has the agent find fallen objects with realistic collision sounds. Additional research has extended these tasks to multiple goals [ 11 ] and to real world agents [ 12 ] . Our task expands upon these works by utilizing a series of real-world sounds, propagated as they would in

Read the complete text