Embodied Hazard Mitigation using Vision-Language Models for Autonomous Mobile Robots
Oluwadamilola Sotomi, Devika Kodi, Kiruthiga Chandra Shekar, Aliasghar Arab · 2025
Read Embodied Hazard Mitigation using Vision-Language Models for Autonomous Mobile Robots by Oluwadamilola Sotomi, Devika Kodi, Kiruthiga Chandra Shekar, Aliasghar Arab (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06768.txt.
Introduction
Mobile robots that spot hazards and act on them using vision-language models are demonstrated by Oluwadamilola Sotomi and three co-authors in a 2025 robotics paper. Robots in changing environments should notice anomalies and report them. Adding proactive mitigation, where the robot responds rather than only observing, improves safety and keeps operations running.
The system combines vision-language models with large language models so the robot can perceive a scene, interpret hazards and conflicts, report them in real time, and respond when possible. Hazardous and conflict states are built directly into the decision framework, each triggering specific mitigation strategies. In user studies with thirty participants, the system detected anomalies with about ninety-one percent accuracy and responded with low latency using an edge computing setup. The authors target urban and environmental anomalies as the proving ground.
Catalog record
| CREATOR | Oluwadamilola Sotomi, Devika Kodi, Kiruthiga Chandra Shekar, Aliasghar Arab |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-06768
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Embodied Hazard Mitigation using Vision-Language Models for Autonomous Mobile Robots Oluwadamilola Sotomi, Devika Kodi, Kiruthiga Chandra Shekar, Aliasghar Arab Abstract Autonomous robots operating in dynamic environments should identify and report anomalies. Embodying proactive mitigation improves safety and operational continuity. This paper presents a multimodal anomaly detection and mitigation system that integrates vision-language models and large language models to identify and report hazardous situations and conflicts in real-time. The proposed system enables robots to perceive, interpret, report, and if possible respond to urban and environmental anomalies through proactive detection mechanisms and automated mitigation actions. A key contribution in this paper is the integration of Hazardous and Conflict states into the robot’s decision-making framework, where each anomaly type can trigger specific mitigation strategies. User studies (n = 30) demonstrated the effectiveness of the system in anomaly detection with 91.2% prediction accuracy and relatively low latency response times using edge-ai architecture. I INTRODUCTION As Autonomous Mobile Robots (AMRs) are increasingly deployed in dynamic and unpredictable environments, the ability to proactively detect and respond to anomalies is critical for ensuring safety and operational continuity [ 1 ] . Traditional AMRs operate with limited situational awareness, often reacting to problems only after they occur, which can lead to safety incidents, operational disruptions, and reduced system reliability [ 2 , 3 ] . A key requirement in autonomous robotics is therefore proactive anomaly detection—robots must not only navigate safely but also anticipate and proactively respond to hazards for mitigation before they escalate into critical situations [ 4 ] . Past work has explored reactive safety systems and obstacle avoidance [ 5 ] , but many approaches remain restricted to predefined scenarios, lacking adaptability to new anomalies or changing environments. The early models relied on rule-based or threshold detection methods [ 6 ] , which did not capture contextual information or provide meaningful responses. More recent machine learning approaches combine visual perception and semantic reasoning to improve anomaly detection [ 7 , 8 , 9 ] . Fig. 1: The AMR detects abnormalities during normal operation and proactively responds through real-time reporting and mitigation planning to ensure safety. Recent advances in large language models (LLMs) and vision language models (VLMs) open new opportunities for intelligent anomaly detection [ 10 , 11 ] . These models process multimodal inputs, understand the context, and generate human-like reasoning about hazards and conflicts [ 12 ] . Previous studies emphasize aligning robot behavior with human safety expectations to support trust in real-world deployments [ 13 , 14 , 15 ] . In particular, VLMs improve AMR perception by integrating visual and textual understanding: Grad-CAM [ 16 ] highlights salient regions, VLFM [ 17 ] allows semantic mapping, and BLIP [ 18 ] grounds image–text pairs for hazard identification. These capabilities have been extended to explainable AI safety frameworks that link anomaly detection with autonomous decision making [ 19 , 20 , 21 ] . To address current limitations, we propose a proactive anomaly detection and mitigation system that generates real-time hazard assessments and automated responses for AMR operations. Using VLFM and LLM, our approach integrates camera-based perception, semantic analysis, and language-based reasoning for contextual anomaly detection. A key innovation is the explicit modeling of two distinct anomaly states, Hazardous and Conflict, each linked to targeted mitigation strategies. This work extends our previous research on explainable safety systems [ 2 , 22 ] with expanded anomaly detection capabilities and a real-time ROS2 implementation using AI models on the edge. Our results demonstrate improved safety, faster response, and more transparent decision-making, underscoring the potential of VLFMs and LLMs to enhance operational reliability of AMRs in dynamic environments. II PROBLEM FORMULATION Autonomous mobile robotic systems operating in dynamic environments must proactively detect and respond to anomalies to ensure safety and operational continuity. We define the proactive anomaly detection and mitigation task as a tuple of perception information from the data in the robot world. \mathcal{T}_{\text{robot}}=\left(\mathcal{S},\mathcal{G},\mathcal{P},\mathcal{A},\mathcal{M},\mathcal{H},\mathcal{L},\mathcal{R},\varepsilon\right) (1) where, \mathcal{S}=(\mathbf{q},\mathbf{v},\mathbf{q}_{\text{env}}) is the state of the robot, with \mathbf{q}\in\mathbb{R}^{n} as the position and orientation of the robot, \mathbf{v}\in\mathbb{R}^{n} as the velocity of the robot, \mathbf{q}_{\text{env}}^{j}\in\mathbb{R}^{n} represents the relative state of an observed environmental object. These objects might represent a hazardous situation from the robot’s point of view and can be defined. \mathcal{G}\equiv\mathbf{q}_{g}\in\mathbb{R}^{n} is the goal in the robot workspace. \mathcal{P}=\pi:[0,T]\rightarrow\mathbb{R}^{n} is the planned trajectory that maps time to robot location and velocities, so that the robot safely transitions from the initial state \mathbf{q}_{0} to \mathbf{q}_{g} while avoiding detected anomalies and maintaining operational safety. \mathcal{A}=\{a_{t}\mid t\in[0,T]\} is the set of anomalies detected during execution, where each a_{t} includes the type of anomaly, severity assessment, the state of the robot, the relative state of the objects, and contextual information through multimodal analysis. \mathcal{M}=\{m_{t}\mid t\in[0,T]\} is the set of mitigation actions triggered in response to detected anomalies, where each m_{t} represents automated responses such as path replanning, emergency stops, or alert notifications. \mathcal{H}\subset\mathcal{W} represents the subset of known hazardous states in which the robot is previously exposed to risk of loss, with \mathcal{W} being the set of all possible world states. \mathcal{L}\subset\mathcal{W} represents the subset of loss states in which accidents or undesirable results occur. \mathcal{R}=\{R_{i}(\mathbf{S}_{h})\mid i\in[1,n]\} is the set of risk functions that map hazardous states to loss probabilities, where each R_{i}(\mathbf{S}_{h})\in[0,1] represents the probability of transitioning from a hazardous state to a specific loss class. \varepsilon\in[0,1] is the anomaly detection effectiveness score that reflects the system’s ability to identify and respond to threats, measured through detection accuracy and response time metrics. The set of safety constraints, anomaly detection requirements, and mitigation protocols can be formalized as a set of operational constraints \Omega_{safety} , which must be satisfied at all times. \Omega_{\text{safety}}=\bigcap_{i\in M}\Omega_{i} (2) where \Omega_{i} represents the constraints imposed by the safety requirement i from the set of governing rules M . For this purpose, which can be modeled in different categories as suggested in [ 23 ] . II-A Hazard as Risk for Loss Any state that has a risk or probability of transitioning to a loss state is a hazardous state. In other words, an autonomous robot at any given time of t\in\mathbb{R}>0 can be represented by the state \mathbf{S} that belongs to the general world \mathbf{S}\in\mathcal{W} , where \mathcal{W} represents the set of all possible states. Even though all feasible world states are not known, this formulates the problem by separating known and unknown subsets of the world. Let us also assume that there exists a subset of Hazardous states, named \mathcal{H}\subset\mathcal{W} where the robot is in a state exposed to risk of loss. Similarly, in practice, all these states are not identified and an unknown subset of hazardous states might exist. Hence, this set is dynamic and, if any loss occurs, the states that ended up in a loss event should be added to \mathcal{H} , the hazardous subset. Remark 1 ( Robot World ) . Representing the robot’s world \mathcal{T}_{\text{robot}} in the universal state space \mathcal{W} and state \mathbf{S} that is dependent on the autonomy architecture, capabilities and limitations of the robot’s perception technology and the context at the scenario. However, for the purposes of formulating this problem, the world as perceived from the autonomous robot’s point of view consists of the spatio-temporal