A Comprehensive Survey on Physical Risk Control in the Era of Foundation Model-enabled Robotics

Takeshi Kojima, Yaonan Zhu, Yusuke Iwasawa, Toshinori Kitamura, Gang Yan, Shu Morikuni, Ryosuke Takanami, Alfredo Solano, Tatsuya Matsushima, Akiko Murakami, Yutaka Matsuo · 2025

Read A Comprehensive Survey on Physical Risk Control in the Era of Foundation Model-enabled Robotics by Takeshi Kojima, Yaonan Zhu, Yusuke Iwasawa, Toshinori Kitamura, Gang Yan, Shu Morikuni, Ryosuke Takanami, Alfredo Solano, Tatsuya Matsushima, Akiko Murakami, Yutaka Matsuo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-12583.txt.

Introduction

Takeshi Kojima and a large team survey physical risk control for robots driven by foundation models. Such robots can handle many tasks and may replace human labour in more settings, yet they act on the physical world, where errors can harm people and objects. The survey sorts control methods across the whole life of a robot system.

Three phases frame the review: before deployment, before any incident, and after an incident. The team finds thin research on risk cuts in the pre-incident phase, on work that assumes physical contact with humans, and on flaws inside foundation models themselves. They hope the map guides safer human and robot work together.

Catalog record

CREATORTakeshi Kojima, Yaonan Zhu, Yusuke Iwasawa, Toshinori Kitamura, Gang Yan, Shu Morikuni, Ryosuke Takanami, Alfredo Solano, Tatsuya Matsushima, Akiko Murakami, Yutaka Matsuo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-12583

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Comprehensive Survey on Physical Risk Control in the Era of Foundation Model-enabled Robotics Takeshi Kojima, Yaonan Zhu, Yusuke Iwasawa, Toshinori Kitamura, Gang Yan, Shu Morikuni, Ryosuke Takanami, Alfredo Solano, Tatsuya Matsushima, Akiko Murakami, Yutaka Matsuo Abstract Recent Foundation Model-enabled robotics (FMRs) display greatly improved general-purpose skills, enabling more adaptable automation than conventional robotics. Their ability to handle diverse tasks thus creates new opportunities to replace human labor. However, unlike general foundation models, FMRs interact with the physical world, where their actions directly affect the safety of humans and surrounding objects, requiring careful deployment and control. Based on this proposition, our survey comprehensively summarizes robot control approaches to mitigate physical risks by covering all the lifespan of FMRs ranging from pre-deployment to post-accident stage. Specifically, we broadly divide the timeline into the following three phases: (1) pre-deployment phase, (2) pre-incident phase, and (3) post-incident phase. Throughout this survey, we find that there is much room to study (i) pre-incident risk mitigation strategies, (ii) research that assumes physical interaction with humans, and (iii) essential issues of foundation models themselves. We hope that this survey will be a milestone in providing a high-resolution analysis of the physical risks of FMRs and their control, contributing to the realization of a good human-robot relationship. † † † Corresponding Authors. 1 Introduction Figure 1: Motivation of our survey. Studies on conventional robotics and Deep Learning-based robotics (before foundation models) were mainly based on closed environments where physical risks are excluded or minimized by restricting the entry of humans and objects, e.g., inside a factory. Foundation Model-enabled robotics (FMRs) are expected to be utilized in an open world where physical risks inevitably exist because humans and robots are always in close proximity and physically interacting, e.g. a restaurant. Hence physical risk control is becoming more important in the era of FMRs. Since the emergence of foundation models, robotics has shown dramatic improvements in general-purpose and highly adaptable manipulation skills, indicating that it has entered a new era called Foundation Model-enabled robotics (FMRs). FMRs leverage large-scale pre-trained neural network models that integrate language, vision, and action modalities, enabling robots to generalize across diverse tasks Firoozi et al. 2023 . They comprise large language models (LLMs) and vision-language models (VLMs) for language comprehension and visual understanding, enhancing high-level task planning for long-horizon tasks Liang et al. 2023 . Additionally, they include robot transformers that integrate perception, decision-making, and action generation to process multimodal inputs and generate low-level motion commands for end-to-end control Brohan et al. 2022 . In contrast to conventional robotic engineering, which generally controls robots based on human-crafted rules, FMRs learn to control themselves from enormous amounts of data with a statistical approach. This paradigm shift has enhanced generalizability and long-horizon reasoning Zawalski et al. 2024 , enabling FMRs to show promising advantages over classical methods in adapting to diverse tasks and unstructured environments. They have been successfully applied to various fields, including task planning Ahn et al. 2022 , vision language guided manipulation O’Neill et al. 2023 , tactile perception Zhao et al. 2024a , locomotion Bohlinger et al. 2024 , and navigation Moroncelli et al. 2024 . ( ➢ Section 2 ) While we could enjoy economic benefits by replacing human labor with robots in various applications, we cannot completely avoid the risk of FMRs causing physical damage to surrounding humans or objects. As FMRs are engaged in more challenging tasks and environments, often requiring contact with people and objects, such as housework, surgery or nursing care, the risk of causing physical damage increases. Of course, even in such an environment, it is possible to mitigate some risks through the establishment of social rules and education, e.g., do not play near robots. However, we cannot completely eliminate accidents because of our misperception of the environment (e.g. robots in blind spots) or unexpected environmental changes (e.g., sudden hardware failures). ( ➢ Section 3 ) Based on the premise that FMRs cannot completely eliminate the risk of causing physical damage, this survey comprehensively summarizes robot control approaches against physical risks by covering all the lifespan of FMRs, from pre-deployment to post-accident stage. Specifically, our study conducts this survey of physical risk controls by dividing the timeline into the following three phases: (1) pre-deployment phase, i.e., risk prevention phase when learning from data, (2) pre-incident phase, i.e., before an incident happens after deployment, and (3) post-incident phase, i.e., the recovery and improvement stage. ( ➢ Figure 3 and Section 4 ) We emphasize that many classic robotic studies discussed safety control within a closed environment where humans could intervene and prevent hazardous incidents, such as inside a factory or laboratory. In such an environment, if a robot causes an incident, pressing the emergency stop button will solve the problem. In contrast, our survey focuses on the safety of FMRs assumed to act in an open world, where bigger physical risks often exist because humans and robots are always in close proximity and physically interacting, e.g. inside a home or a cafe. In this case, we need to consider how to recover or treat robots as well as surrounding humans and objects because “life still goes on” for both robots and humans after the incident. We also emphasize that prior surveys of FMRs focused on only some partial sections within the first two phases of safety control, i.e., pre-deployment and pre-incident phase in light of our classification scheme, and so have been insufficiently organized in detail Bommasani et al. 2021 ; Hu et al. 2023 ; Firoozi et al. 2023 ; Xiao et al. 2023 . In other words, they only summarized partial sections of the mitigation strategies of physical risks before an incident, and generally have not covered post-incident recovery actions. ( ➢ Figure 2 ) Throughout this survey, we have found that there is much room to study (i) pre-incident risk mitigation strategies, (ii) research that assumes physical interaction with humans, and (iii) essential issues of foundation models themselves. We emphasize that social measures such as legislation or insurance schemes are also important aspects to enhance mitigation of physical damage. ( ➢ Section 5 ) Figure 2: Scope of our survey. FMRs are expected to be utilized in an open world where humans and robots are interacting with each other. Based on this proposition, our study conducts a comprehensive suvey on physical risk control by categorizing the lifespan of FMRs into (1) pre-deployment phase, (2) pre-incident phase after deployment, and (3) post-incident phase. In contrast, prior surveys of FMRs mainly focused on partial sections of the first two phases. 2 Robotics and Foundation Models 2.1 Conventional Robotics Engineering Conventional robotics has driven innovation for long, predating the rise of data-driven techniques. While modern robotics increasingly rely on deep learning and large-scale data, classical theories and engineering remain essential. In particular, it focuses on five pivotal directions: (1) innovative mechanical design Sugiura et al. 2024 , (2) precise motion control Sun et al. 2018 , (3) advanced perception systems Mahler et al. 2019 , (4) planning and decision-making strategies La Valle 2011 ; La Valle 2011 ; Hansel et al. 2023 , (5) adaptive learning and optimization methodologies Zhang et al. 2024 ; Saveriano et al. 2023

Read the complete text