Human Interaction for Collaborative Semantic SLAM using Extended Reality

Laura Ribeiro, Muhammad Shaheer, Miguel Fernandez-Cortizas, Ali Tourani, Holger Voos, Jose Luis Sanchez-Lopez · 2025

Read Human Interaction for Collaborative Semantic SLAM using Extended Reality by Laura Ribeiro, Muhammad Shaheer, Miguel Fernandez-Cortizas, Ali Tourani, Holger Voos, Jose Luis Sanchez-Lopez (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14949.txt.

Introduction

Laura Ribeiro and five co-authors bring human help into robot mapping through extended reality in a 2025 robotics paper. Semantic SLAM systems build maps labeled with rooms and structures while tracking the robot's position. Real sites defeat them with occlusions, missing data, and ambiguous shapes, while humans grasp the layout at a glance.

Their system, HICS-SLAM, gives operators a shared extended reality view of the robot's three-dimensional scene graph. People can inspect the map live and add high-level concepts such as rooms directly into it. A graph-based fusion method merges human input with robot perception in a scalable way. Trials on real construction site data show better room detection and scene understanding, pointing toward mapping teams where humans and robots correct each other.

Catalog record

CREATORLaura Ribeiro, Muhammad Shaheer, Miguel Fernandez-Cortizas, Ali Tourani, Holger Voos, Jose Luis Sanchez-Lopez
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-14949

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Human Interaction for Collaborative Semantic SLAM using Extended Reality Laura Ribeiro, Muhammad Shaheer, Miguel Fernandez-Cortizas, Ali Tourani, Holger Voos, Jose Luis Sanchez-Lopez Abstract Semantic SLAM (Simultaneous Localization and Mapping) systems enrich robot maps with structural and semantic information, enabling robots to operate more effectively in complex environments. However, these systems struggle in real-world scenarios with occlusions, incomplete data, or ambiguous geometries, as they cannot fully leverage the higher-level spatial and semantic knowledge humans naturally apply. We introduce HICS-SLAM, a Human-in-the-Loop semantic SLAM framework that uses a shared extended reality environment for real-time collaboration. The system allows human operators to directly interact with and visualize the robot’s 3D scene graph, and add high-level semantic concepts (e.g., rooms or structural entities) into the mapping process. We propose a graph-based semantic fusion methodology that integrates these human interventions with robot perception, enabling scalable collaboration for enhanced situational awareness. Experimental evaluations on real-world construction site datasets demonstrate improvements in room detection accuracy, map precision, and semantic completeness compared to automated baselines, demonstrating both the effectiveness of the approach and its potential for future extensions. I Introduction Simultaneous Localization and Mapping (SLAM) constructs environmental maps while estimating robot position [ 1 ] . Traditional approaches focus on geometric representations, while Semantic SLAM enriches maps with structural and semantic information, enabling systems to identify objects, entities, and their associations beyond geometry [ 2 , 3 ] . However, semantic mapping remains sensitive to noise and misclassified elements, which propagate errors and reduce map robustness [ 4 ] . These limitations affect applications requiring accurate spatial understanding, including search-and-rescue operations, industrial assembly [ 5 , 6 ] , and construction sites. To address these challenges, Human-in-the-Loop (HitL) strategies incorporate human knowledge into the mapping process. Fig. 1 : The overall idea behind HICS-SLAM: a human interacts with the map through the proposed extended reality application in a virtual world, adding new rooms (characterized by red circles) to the generated 3D scene graph and updating them in real time. These modifications propagate through the SLAM algorithm, enhancing mapping accuracy and localization performance simultaneously. HitL methodologies enable interactive correction of trajectory estimation and semantic labeling during SLAM operation [ 7 ] . Recent works, like HAC-SLAM [ 8 ] , introduced trajectory corrections in the SLAM, while HSS-SLAM [ 9 ] used superquadric representations for map refinement. However, the methods could be improved using high-level semantic editing, since HAC-SLAM lacks semantic integration into scene graphs, and HSS-SLAM requires manual parameter adjustments that are not intuitive for non-experts and cannot support room-scale semantic editing. We introduce HICS-SLAM, a new approach that combines HitL features with Semantic SLAM through a virtual environment. HICS-SLAM enables high-level semantic user operations, such as creating new rooms in the scene graph, through an Extended Reality (XR) shared environment. XR interfaces provide intuitive means to inspect reconstructed environments and perform corrections through natural hand gestures [ 10 , 11 ] , offering advantages over 2D interfaces through spatially grounded interactions that leverage human spatial cognition [ 12 ] . HICS-SLAM uses scene graphs, which represent environments hierarchically as nodes and edges, allowing SLAM systems to optimize objects and rooms [ 2 ] . This hierarchical representation enables interventions at the graph level, allowing corrections to propagate across related elements while maintaining global semantic and geometric consistency [ 13 ] . Combining XR with a 3D scene graph, HICS-SLAM allows users to inspect reconstructed environments and define semantic structures through hand gestures, which are integrated into the S-Graphs 2.0 [ 13 ] backend and propagate through the SLAM system. Figure 1 shows a human operator utilizing the HICS-SLAM system. This workflow supports real-time data collection, SLAM processing, and interactive human refinement, shifting manual corrections from post-processing to real-time while improving map quality and localization consistency. The main contributions of this work are: • A collaborative SLAM framework that enables intuitive human intervention through XR interfaces, allowing direct manipulation of 3D scene graphs for spatial modeling. • An online integration method that incorporates human semantic knowledge into the SLAM back-end optimization, improving mapping quality through high-level corrections. • Real-world experimental validation demonstrating the effectiveness of human-guided semantic corrections for improving SLAM applications. II Related Works HitL strategies have become powerful approaches for combining human interaction with automated mapping and navigation systems [ 1 ] . Early methods employed factor graph extensions to allow joint optimization of trajectories and maps [ 1 ] , while subsequent systems provided mechanisms for interactive trajectory correction, such as HAC-SLAM [ 8 ] , and semantic-assisted mapping, as in A-SLAM [ 7 ] . More recent frameworks incorporate semantic representations to facilitate higher-level interventions, including HSS-SLAM, which models environments with superquadrics [ 9 ] , Collaborative Human Augmented SLAM, enabling multi-user semantic collaboration [ 14 ] , and approaches that automatically infer semantic-relational structures from factor graphs [ 15 ] . Evaluations of topological mapping methods in indoor environments [ 4 ] highlight challenges that remain for automated systems, motivating human intervention approaches. Extended Reality methodologies have enhanced human-robot collaboration by providing intuitive interfaces for map inspection and manipulation. Direct SLAM applications include Interactive 3D Graph SLAM [ 16 ] , Mixed Reality (MR) Robot Control [ 17 ] , and 3D MR interfaces for human-robot teaming [ 18 ] , which enable real-time 3D visualization and map corrections. Related collaborative navigation approaches include intuitive human-drone navigation in unknown environments through mixed reality [ 19 ] and spatial-assisted human-drone collaborative frameworks [ 20 ] . Broader applications demonstrate XR potential in robotics: HoloDesigner [ 21 ] for on-site design, mixed-reality teleoperation for high-level control of legged-manipulator robots [ 22 ] , Boston Dynamics Spot integration with Microsoft HoloLens 2 (MR Device) [ 23 ] for robot teleoperation, MR integration with Building Information Modeling (BIM) [ 24 ] , digital twin frameworks for interactive building operations [ 25 , 12 ] , and comprehensive XR approaches for enhanced human-robot collaboration [ 26 ] . Despite these advances, most existing HitL approaches either operate at low-level geometric representations or provide high-level interaction without incorporating semantic information, making manual intervention time-consuming, prone to inaccuracies, and less effective in complex or large-scale environments. Our approach bridges this gap by enabling high-level semantic map interaction in XR, combining intuitive scene manipulation with semantic awareness to support scalable human-robot collaboration. III System Overview Fig. 2 : System architecture overview. The proposed method enables real-time collaboration between the robot and human through a shared virtual environment. The robot operates in the real world while generating a hierarchical scene graph representation, and humans interact with this virtual representation using the HoloLens 2 device and hand gestures to enhance semantic understanding, creating a bidirectional knowledge exchange between real and virtual worlds. III-A Virtual World, Real World: Collaborative Mapping Our system enables collaborative mapping where a robot operates in the physical environment and collaborates with a human operator through a shared virtual representation of the robot’s situational awareness. The robot collects sensor data from its environment, constructing a hierarchical world representation that encompasses geometric primitives (planes, keyframes) and semantic concepts (rooms, walls, objects). This knowledge is accessible to the human operator through an MR device that renders an interactive virtual environment. The human

Read the complete text