Vision-Based Safe Human-Robot Collaboration with Uncertainty Guarantees

Jakob Thumm, Marian Frei, Tianle Ni, Matthias Althoff, Marco Pavone · 2026

Read Vision-Based Safe Human-Robot Collaboration with Uncertainty Guarantees by Jakob Thumm, Marian Frei, Tianle Ni, Matthias Althoff, Marco Pavone (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15221.txt.

Introduction

Thumm and colleagues build a vision-based safety pipeline for human-robot collaboration with statistical guarantees. Their system predicts 3D human motion with conformal bounds on prediction error, replacing out-of-distribution poses with past predictions to keep motion smooth. The prediction sets feed directly into the provably safe SARA shield. Tests on the Human3.6M collection and a live joint setup show the sets run more than seven times smaller in volume than a baseline, meeting a necessary condition for performance level d.

Catalog record

CREATORJakob Thumm, Marian Frei, Tianle Ni, Matthias Althoff, Marco Pavone
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23867
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15221

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Vision-Based Safe Human-Robot Collaboration with Uncertainty Guarantees Jakob Thumm, Marian Frei, Tianle Ni, Matthias Althoff, Marco Pavone Abstract We propose a framework for vision-based human pose estimation and motion prediction that gives conformal prediction guarantees for certifiably safe human-robot collaboration. Our framework combines aleatoric uncertainty estimation with OOD detection for high probabilistic confidence. To integrate our pipeline in certifiable safety frameworks, we propose conformal prediction sets for human motion predictions with high, valid confidence. We evaluate our pipeline on recorded human motion data and a real-world human-robot collaboration setting. I Introduction Autonomous robots will become an integral part of our society, performing tedious and dangerous tasks in industry, households, and healthcare. To ensure safety in these human-centered environments, it is crucial to accurately perceive human poses, predict their motion, and control the robot to prevent critical collisions with the human. Hereby, we have to provide certifiable safety guarantees in all possible unseen situations. Current safe human-robot collaboration (HRC) approaches can provably guarantee human safety if an accurate human pose measurement is available [ 1 , 2 , 3 , 4 ] . These approaches typically rely on a marker-based motion-tracking system for accurate pose estimation, which drastically limits their deployment potential. Previous works that estimate the human pose from more mobile sensor modalities, e.g., RGB-D cameras, usually assume a fixed maximal estimation error [ 5 , 6 , 7 , 8 , 9 ] or provide uncertainty estimates without conformal guarantees [ 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 ] . Most of these approaches can catastrophically fail if inputs are out of distribution (OOD), which is not captured in the predicted aleatoric uncertainty. For predicting all possible states that the human can occupy in a given time interval, traditional safe HRC approaches [ 5 , 6 , 7 , 8 , 9 , 2 , 3 , 4 ] use simple motion models, e.g., that the human can move with up to v_{\text{max}}= 1.6\text{\,}\mathrm{m}\text{\,}{\mathrm{s}}^{-1} in any direction as defined in ISO 13855:2010 [ 19 ] . These models tend to be highly conservative as they are not data-driven. State-of-the-art human motion prediction models [ 20 , 21 , 22 , 23 , 24 , 25 ] are less conservative but often lack (i) heteroscedastic aleatoric uncertainty estimates, (ii) end-to-end propagation of the pose estimation uncertainty, and (iii) conformal prediction guarantees. To alleviate these shortcomings, we propose a framework for vision-based human pose estimation and motion prediction in Fig. 1 . First, we estimate the two-dimensional (2D) human pose and its covariances in the two camera frames. Then, we perform an uncertainty-aware triangulation to retrieve the 3D pose with covariances. Given a history of human poses, we predict future 3D poses and their covariances. Based on the predicted covariances, we propose conformal prediction sets to over-approximate the uncertainty in the motion prediction with an 1-\epsilon confidence bound. To detect critical OOD inputs in the pose estimation and motion prediction, we use the gradient-based OOD detection method in [ 26 ] . By reusing past motion predictions as potential replacements for OOD inputs, we establish a smooth operation of our pipeline in unseen situations. The conformal prediction sets returned by our pipeline directly feed into the provably safe HRC approach SARA shield [ 4 ] , which establishes certifiable safety guarantees at all times. The main contribution of our work is a framework for safe HRC from vision-based inputs featuring • 3D human motion prediction with end-to-end uncertainty propagation, • conformal prediction sets for human poses, and • a method to handle OOD inputs in a continuous prediction setting. We evaluate our pipeline on the Human3.6M dataset [ 27 ] and in a real-world HRC setting with SARA shield [ 4 ] . In our experiments, we find that (i) our human pose estimation and motion prediction performs similarly to state-of-the-art models, (ii) the conformal prediction sets reduce the conservatism of model-based predictions by a factor of 11 , and (iii) our proposed OOD handling reduces interruptions in real-world operation by 36.0\text{\,}\mathrm{\%} . Fig. 1 : Methodological overview of our pose estimation and motion prediction pipeline with conformal prediction sets. II Preliminaries For brevity, we use the colon notation A:B to refer to the sequence A,A+1,\dots,B . We denote a point in the two-dimensional (2D) image space as (u,v)\in\mathbb{R}^{2} and a point in the three-dimensional (3D) Euclidean space as \boldsymbol{p}\in\mathbb{R}^{3} . The human pose at timestep k is denoted by a tuple of 3D points \mathcal{P}_{k}=(\boldsymbol{p}^{1}_{k},\boldsymbol{p}^{2}_{k},\dots,\boldsymbol{p}^{J}_{k})\in\mathbb{R}^{J\times 3} , and the motion of a human at the discrete timesteps 0,1,\dots K is denoted by a sequence of poses \mathcal{P}_{1:K}=(\mathcal{P}_{1},\mathcal{P}_{2},\dots,\mathcal{P}_{K})\in\mathbb{R}^{K\times J\times 3} . Finally, we refer to a sphere with center \boldsymbol{p}_{c} and radius r as \mathcal{B}(\boldsymbol{p}_{c},r) . II-A Conformal Predictions Let \mathcal{Z}={(\boldsymbol{x}_{i},\boldsymbol{y}_{i})}_{i=1:N} be a set of i.i.d. training tuples with inputs \boldsymbol{x}_{i}\in\mathcal{X} and labels \boldsymbol{y}_{i}\in\mathcal{Y} , which is devided into a training set \mathcal{Z}^{\text{train}} and a calibration set \mathcal{Z}^{\text{cal}} with \mathcal{Z}^{\text{train}}\cup\mathcal{Z}^{\text{cal}}=\mathcal{Z} . Further, let a prediction model f_{\theta}:\mathcal{X}\rightarrow\mathcal{Y} be trained on \mathcal{Z}^{\text{train}} . We denote a bounded set \mathcal{S}_{i}\subseteq\mathcal{Y} as a conformal prediction set if P(\boldsymbol{y}_{i}\in\mathcal{S}_{i})\geq 1-\epsilon with confidence level 1-\epsilon [ 28 , Sec. 2.3] . A non-conformity measure A(\mathcal{Z}^{\text{train}},z_{i})\in\mathbb{R} is a function that scores how dissimilar a calibration sample z_{i} is from the training dataset [ 29 ] . II-B Problem Statement Given a history of N_{I} stereo image pairs \{(\mathbf{I}_{k,1},\mathbf{I}_{k,2})\}_{k=1:N_{I}} with f_{\text{cam}}=\tfrac{1}{t_{k+1}-t_{k}} frames per second, our goal is to predict a sequence of N_{P} tuples of J conformal prediction sets ((\mathcal{S}^{1}_{1},\mathcal{S}^{2}_{1},\dots\mathcal{S}^{J}_{1}),\dots,(\mathcal{S}^{1}_{N_{P}},\mathcal{S}^{2}_{N_{P}},\dots\mathcal{S}^{J}_{N_{P}})) that include the future human joint positions with 1-\epsilon confidence. III Human Pose Pipeline To solve the problem in Section II-B , we propose the human pose pipeline depicted in Fig. 1 . III-A Uncertainty-Aware 3D Pose Estimation For our 3D pose estimation, we first adapt YOLO26 [ 30 ] to return the covariance matrix \boldsymbol{C}^{j}_{i,\text{2D}}\in\mathbb{R}^{2\times 2} in addition to the 2D mean (u_{i}^{j},v_{i}^{j}) of each human joint j=1,\dots,J in the two calibrated camera images \mathbf{I}_{1} and \mathbf{I}_{2} . We extract the approximated covariance matrix from the scale of the Gaussian distribution returned by the regression model of the flow-based regression [ 11 ] used in YOLO26. We then estimate the 3D joint positions from the two 2D poses by standard linear triangulation [ 31 ] . Following [ 32 ] , we estimate the cross-covariance from residual correlations over the calibration set \mathcal{Z}^{\text{cal}} . We then obtain the 3D covariances through first-order covariance propagation [ 33 ] and add a small constant isotropic term to the covariances to account for systematic reconstruction errors, e.g., camera calibration imperfections. III-B Uncertainty-Aware 3D Motion Prediction We refer to the tuple of covariance matrices of a human pose at timestep k as \mathcal{C}_{k}=(\boldsymbol{C}^{1}_{k},\boldsymbol{C}^{2}_{k},\dots,\boldsymbol{C}^{J}_{k})\in\mathbb{R}^{J\times 3\times 3} . Given a sequence of past poses \mathcal{P}_{1:K_{I}} and their covariances \mathcal{C}_{1:K_{I}}=(\mathcal{C}_{1},\mathcal{C}_{2},\dots,\mathcal{C}_{K_{I}}) , we predict a sequence of future human 3D poses \mathcal{P}_{K_{I}+1:K_{I}+K_{P}} and their heteroscedastic aleatoric covariance matrices \mathcal{C}_{K_{I}+1:K_{I}+K_{P}} . III-B 1 Model Architecture For our motion prediction model, we extend the discrete cosine transform (DCT) transformer model of [ 21 ] with uncertainty inputs and outputs. We first apply a DCT to the pose and covariance inputs \mathcal{P}_{1:K_{I}} and \mathcal{C}_{1:K_{I}} along the temporal axis

Read the complete text