Enhancing Human-Robot Collaboration: A Sim2Real Domain Adaptation Algorithm for Point Cloud Segmentation in Industrial Environments

Fatemeh Mohammadi Amin, Darwin G. Caldwell, Hans Wernher van de Venn · 2025

Read Enhancing Human-Robot Collaboration: A Sim2Real Domain Adaptation Algorithm for Point Cloud Segmentation in Industrial Environments by Fatemeh Mohammadi Amin, Darwin G. Caldwell, Hans Wernher van de Venn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-09552.txt.

Introduction

Fatemeh Mohammadi Amin, Darwin Caldwell, and Hans Wernher van de Venn help robots read factory floors in this 2025 paper. Safe teamwork between people and robots needs a precise map of the scene, and learning such maps takes labeled 3D data that is scarce in real plants.

Their method trains on simulated point clouds and adapts to real ones, a sim-to-real jump. The network joins two streams, one graph based and one image based, with residual layers. On real human-robot setups and simulated plant clouds it reached 97.76 percent segmentation accuracy, beating past state of the art results and holding up well across the sim-to-real gap.

Catalog record

CREATORFatemeh Mohammadi Amin, Darwin G. Caldwell, Hans Wernher van de Venn
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-09552

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing Human-Robot Collaboration: A Sim2Real Domain Adaptation Algorithm for Point Cloud Segmentation in Industrial Environments Fatemeh Mohammadi Amin, Darwin G. Caldwell, Hans Wernher van de Venn Abstract The robust interpretation of 3D environments is crucial for human-robot collaboration (HRC) applications, where safety and operational efficiency are paramount. Semantic segmentation plays a key role in this context by enabling a precise and detailed understanding of the environment. Considering the intense data hunger for real-world industrial annotated data essential for effective semantic segmentation, this paper introduces a pioneering approach in the Sim2Real domain adaptation for semantic segmentation of 3D point cloud data, specifically tailored for HRC. Our focus is on developing a network that robustly transitions from simulated environments to real-world applications, thereby enhancing its practical utility and impact on a safe HRC. In this work, we propose a dual-stream network architecture (FUSION) combining Dynamic Graph Convolutional Neural Networks (DGCNN) and Convolutional Neural Networks (CNN) augmented with residual layers as a Sim2Real domain adaptation algorithm for an industrial environment. The proposed model was evaluated on real-world HRC setups and simulation industrial point clouds, it showed increased state-of-the-art performance, achieving a segmentation accuracy of 97.76%, and superior robustness compared to existing methods. keywords human-robot collaboration (HRC), Semantic Segmentation, Domain Adaptation, Real-time Segmentation 1 Introduction Safe human-robot collaboration (HRC) is becoming increasingly important under the Industry 4.0 paradigm, which emphasizes intelligent and flexible automation. Traditional robots are being replaced by collaborative robots (cobots), designed for safe and efficient interaction with humans in dynamic industrial settings. In fact, cobots are increasingly being used for flexible task accomplishment instead of traditional industrial robots Olender and Banas 2019 and can work in the same workspace as humans Vicentini 2021 . To effectively collaborate with humans, cobots need advanced intelligence, allowing them to respond to human inputs and adapt to dynamic environments for smooth, safe, and productive workflows Mohammadi Amin et al. 2020 . This requires advanced sensing and perception capabilities Hamon et al. 2020 , like semantic segmentation of 3D point cloud data, which is vital for accurately understanding objects in industrial environments and ensuring safety by detecting potential hazards Milioto et al. 2019 ; Guo et al. 2018 . Despite the advances in machine learning techniques, deploying these models in industrial settings poses significant challenges. This is particularly true for tasks that involve humans, such as human-robot collaboration, where ensuring human safety is crucial Mohammadi Amin et al. 2020 . Factors such as data hunger—a term referring to the performance limitations due to insufficient datasets-, restrictions on using realistic datasets, and the time-consuming nature of manual annotation prevent the acquisition of large-scale, labeled point cloud datasets, which are necessary for training accurate models Gao et al. 2020 . To address these challenges, transfer learning and domain adaptation techniques, which enable models to generalize across diverse datasets and domains, are increasingly being explored Triess et al. 2021 . This paper addresses the urgent issue of semantic segmentation of 3D point clouds within HRC environments. It achieved this by leveraging Sim2Real domain adaptation techniques, where the use of real-world data is not completely feasible due to safety concerns and practical limitations such as equipment availability constraints like the lack of diverse robots, and data collection difficulties. Sim2Real domain adaptation refers to strategies aimed at adapting a model trained in a simulated environment to perform efficiently in the real world. Therefore, the utilization of simulation data as an alternative means to train robust and efficient semantic segmentation models was explored. This leveraging of simulation data not only overcomes the lack of annotated real-world datasets but also provides a controlled environment to generate diverse and challenging scenarios for comprehensive model training. One of the primary challenges associated with employing simulation data for training semantic segmentation models lies in achieving data realism and accuracy. By leveraging cutting-edge simulation tools, it is possible to generate synthetic 3D point clouds that closely mimic industrial environments, while maintaining control over crucial aspects such as object interactions, lighting conditions, and material properties. A two-stream network called FUSION is leveraged in the methodology for effective segmentation. This approach combines two distinct neural network architectures to enhance performance, optimizing the system for more accurate segmentation tasks. This network is a crucial component of the proposed approach. Initially, training is conducted using simulated data. Subsequently, the network is refined and adapted through real-world data applications, ensuring its robust performance and accurate interpretation within the dynamic environments of real-world industrial settings. The results showcase enhanced generalizability and performance in Human-Robot Collaboration environments, achieving 97.6 percent accuracy in real-world settings and 98.9 percent in simulation scenarios. The results also underscore the potential of this approach to contribute substantially to human safety in industrial Human-Robot Collaboration (HRC) settings by ensuring accurate and reliable identification of human and related safety hazards. The main contributions of this paper are as follows: • A domain adaptation algorithm is presented to segment a 3D sparse point cloud. • The dataset for domain adaptation, containing both, real and simulation data, is published. • This is expected to be the first domain adaptation algorithm for real-time and real HRC applications, with testing conducted in real-world laboratory setups. The rest of this paper is laid out as follows. Section II provides a review of related work in the areas of semantic segmentation of 3D point cloud data and Sim2Real domain adaptation. Section III describes the proposed methodology, while Section IV presents the experimental results. Finally, Section V offers a discussion of the findings and implications, along with potential directions for future research. Figure 1 : The Overal concept of FUSION, Simulation and Real-world Dataset 2 Related Works 2.1 Semantic Segmentation of point clouds: Techniques and Challenges for HRC Semantic segmentation of 3D LiDAR (Light Detection and Ranging) point clouds, which are often sparse and disordered Wang et al. 2020 , aims to assign a semantic label to every point. The most common strategies Guo et al. 2020 for this task are projection based Yang and Wang 2019 ; Ma et al. 2017 ; Wu et al. 2019 ; Milioto et al. 2019 , voxel-based Zhou and Tuzel 2017 ; Meng et al. 2018 , and point-based Zhang et al. 2019 ; Huang et al. 2018 ; Landrieu and Simonovsky 2018 methods. Deep learning has shown significant progress in research on this topic, particularly with PointNet Qi et al. 2016 and PointNet++ Qi et al. 2017 architectures that directly process raw point cloud data. Further advancements like PointCNN Li et al. 2018 and DGCNN Wang et al. 2018 introduced convolution-like operations, enhancing local structures and geometrical capture. However, training of these models mainly relies on ideal simulated data, hence their real-world applicability can be limited due to the domain shift. Point cloud representation, which maintains the original geometric information in 3D space without discretization, has shown substantial interest in real-world applications, including autonomous driving Aksoy et al. 2019 ; Wu et al. 2017 ; Zeng et al. 2018 ; Cortinhal et al. 2020 ; Cheng et al. 2021 ; Thomas et al. 2019 ; Milioto et al. 2019 , navigation Feng et al. 2018 and robotics Zha et al. 2020 applications. Directly converting point clouds into sparse vector representations

Read the complete text