Probing Collision Grounding in Vision-Language Models for Safe Human–Robot Collaboration

Jun Wang, Xiaohao XuXiaonan HuangUniversity of Michigan, Ann Arbor · 2026

Read Probing Collision Grounding in Vision-Language Models for Safe Human–Robot Collaboration by Jun Wang, Xiaohao XuXiaonan HuangUniversity of Michigan, Ann Arbor (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-31196.txt.

Introduction

Jun Wang, Xiaohao Xu and Xiaonan Huang benchmark whether vision-language models can judge robot collision risk from what they see. Their TouchSafeBench holds 2,940 simulated indoor episodes with multi-view depth, trajectory maps and simulator contact labels, testing current-state classification and imminent-collision warnings. The best mean Macro-F1 score remains under 50 percent, and depth cues do not automatically become body-collision evidence. Visual fluency, they conclude, does not imply physical accountability.

Catalog record

CREATORJun Wang, Xiaohao XuXiaonan HuangUniversity of Michigan, Ann Arbor
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81044
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.31196

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Probing Collision Grounding in Vision-Language Models for Safe Human–Robot Collaboration Jun Wang, Xiaohao XuXiaonan HuangUniversity of Michigan, Ann Arbor Abstract Safe human–robot collaboration requires more than visual description: a monitor must determine whether the robot body is safely separated, already colliding with the scene or a person, or about to collide. We call this capability collision grounding : binding visual observations to robot body geometry, camera viewpoint, scene layout, human proximity, and temporal motion in order to infer present and imminent contact. We introduce TouchSafeBench, a physics-grounded benchmark for evaluating collision grounding in vision-language models (VLMs). Built in Habitat 3.0, TouchSafeBench contains 2,940 simulated indoor co-presence episodes across social navigation and social rearrangement, with synchronized multi-view RGB-D observations, top-down trajectory maps, calibrated camera metadata, and simulator-derived contact labels. We study two deployment-facing tasks: classifying the current safety state and warning about imminent collision before contact. Across three frontier or robotics-oriented VLMs and nine visual representations, current models remain far from reliable: the best average Macro-F1 stays below 50%, explicit depth is not automatically transformed into robot-body collision evidence, and robot–scene contact is consistently harder than human-contact risk. TouchSafeBench reveals a central limitation of advanced VLMs (including GPT-5.5, Gemini 3.1 Pro, and Gemini Robotics-ER 1.6): visual fluency does not imply physical accountability. Reliable robot safety monitors will need representations that explicitly bind viewpoint, robot morphology, metric geometry, and future collision. We will release the benchmark upon acceptance. Keywords: Human-Robot Collaboration, Robot Safety, Vision-Language Models, Collision Grounding 1 Introduction Vision-language models (VLMs) are increasingly used as perceptual and reasoning modules for robots, with progress in spatial understanding, embodied planning, and language-conditioned perception [ 1 , 36 , 21 ] . In shared indoor spaces, however, the safety-critical question is not simply whether a frame contains a person, an obstacle, or a dangerous-looking configuration. It is whether the robot’s own body is safely separated, already colliding with the human or the environment, or approaching collision. We call this the collision-grounding gap : the failure to bind visual evidence to robot footprint, camera placement, depth, occlusion, and temporal approach. Classical HRI safety standards (ISO 10218, ISO/TS 15066) provide indispensable but deliberately local and reactive control-level safeguards [ 8 , 9 ] . They do not answer the visual question a VLM-based safety monitor would face: is this short multi-view clip evidence of safe motion, scene collision, human collision, or imminent collision? Existing safety, anomaly, and embodied VLM benchmarks also leave this question under-specified. Many rely on human annotations, single-view RGB, retrospective labels, or semantic danger narratives [ 28 , 10 , 29 , 41 , 23 , 37 ] ; they rarely isolate robot-induced collision under deterministic physical ground truth. We introduce TouchSafeBench, a simulated, physics-grounded benchmark for collision-grounded human–robot safety. Habitat 3.0 provides photorealistic HSSD-HAB scenes, SMPL-X humanoids, rigid-body contact signals, and synchronized RGB-D rendering [ 27 ] . Each rollout records four calibrated views, a top-down video, trajectories, and camera parameters. Labels and diagnostic reference facts are derived from simulator state rather than subjective annotation, allowing the same event to be evaluated across viewpoints and geometric representations. TouchSafeBench evaluates two complementary capabilities. Safety Event Classification asks whether a short clip currently shows safe operation, robot–scene collision, or robot–human collision. Collision Early Warning asks whether scene or human collision is imminent before contact occurs, while avoiding false alarms on close but collision-free near misses. Our experiments ablate modality, viewpoint, and model family. The result is a sharp finding with constructive implications: current VLMs are often visually plausible but physically under-grounded, especially for robot–environment collisions that require connecting camera evidence to robot morphology and contact state. Our contributions are: (i) TouchSafeBench, a repeatable HRI safety benchmark with simulator-derived labels for robot-induced collision risk; (ii) two tasks that separate current-state monitoring from anticipatory warning; and (iii) a controlled empirical analysis showing that geometry, viewpoint, and robotics-oriented pretraining help only when they are tied to collision-grounded representations. 2 Related Work Robot safety and HRI simulation. Robot safety standards such as ISO 10218 and ISO/TS 15066 define core safeguards for speed, force, separation monitoring, protective stops, and emergency stopping [ 8 , 9 ] . These safeguards are essential for deployed systems, but they operate mainly through local sensing and control constraints; they do not evaluate multi-view visual evidence or infer whether a near-contact scene is becoming unsafe. Embodied simulators provide a complementary route for repeatable evaluation. AI2-THOR, ThreeDWorld, iGibson, VirtualHome, and Habitat support interactive indoor agents, object manipulation, or humanoid simulation [ 12 , 3 , 14 , 26 , 27 ] . We build on Habitat 3.0 because it combines photorealistic indoor scenes, SMPL-X humanoids, and rigid-body contact signals, enabling controlled tests of robot-induced collision risk. Table 1: Comparison with related safety benchmarks. TouchSafeBench uniquely combines HRI co-presence, multi-view RGB-D, simulator contact labels, and anticipatory warning; the table is intended to compare safety-oriented benchmark families rather than all embodied-AI benchmarks. Benchmark family HRI Views Depth Physics Warn What is primarily tested? ASIMOV-1.0/2.0 \times \times \times \times \times broad semantic physical danger Safety QA benchmarks \times partial \times partial \times safety QA or safe action choice ResponsibleRobotBench partial ✓ ✓ \times \times action responsibility under visual context Embodied VLM benchmarks partial partial \times \times \times embodied perception, planning, and instruction following Video anomaly datasets \times \times \times \times \times retrospective video anomaly detection TouchSafeBench ✓ ✓ ✓ ✓ ✓ robot-induced contact and pre-contact HRI risk Safety, anomaly, and embodied VLM benchmarks. Semantic safety benchmarks such as ASIMOV-1.0/2.0, SafeAgentBench, IS-Bench, AGENTSAFE, ResponsibleRobotBench, and SafeMind evaluate whether foundation models recognize unsafe situations or reason about safe actions [ 28 , 10 , 39 , 22 , 40 , 44 , 2 ] . They are valuable for broad physical danger understanding, but their labels are typically human-authored, rule-based, or scenario-level, and the unsafe agent is often not a robot physically co-present with a human. Video anomaly detection (VAD) benchmarks and VLM-based VAD methods detect retrospective abnormal events from surveillance videos [ 29 , 31 , 24 , 41 , 33 , 42 , 43 , 38 , 15 ] ; they do not target robot body contact, calibrated depth, or pre-contact HRI warning. General embodied VLM benchmarks such as RoboBench, EmbodiedBench, and ERIQ evaluate perception, planning, and interaction competence [ 23 , 37 , 18 ] , but do not isolate collision state and imminent collision risk with simulator-derived labels. Table 1 summarizes related safety benchmark families and adjacent safety-relevant evaluations. TouchSafeBench is deliberately narrower, but the narrowness is the point—it tests the collision-grounded safety capability that broad semantic benchmarks leave entangled with annotation style and scenario description. Geometric and embodied VLM reasoning. Recent VLMs increasingly incorporate geometry through depth images, top-down layout, 3D point representations, and robotics-oriented multimodal training [ 1 , 36 , 20 , 17 , 34 , 45 , 11 , 13 , 30 ] . Yet safety-critical evaluations continue to reveal perception-to-physics misalignment: models may describe a scene plausibly while failing to ground that description in action consequences, spatial thresholds, or contact constraints [ 32 ] . TouchSafeBench turns this concern into a measurable question: when the same physical event is shown through different views and modalities, can a VLM bind the observation

Read the complete text