SPIRIT : Perceptive Shared Autonomy for Robust Robotic Manipulation under Deep Learning Uncertainty
Jongseok Lee 1 4 , Ribin Balachandran 1 , Harsimran Singh 1 , Jianxiang Feng 1 , Hrishik Mishra 1 , Marco De Stefano 1 , Rudolph Triebel 1 2 , Alin Albu-Schäffer 1 3 and Konstantin Kondak 1 · 2026
Read SPIRIT : Perceptive Shared Autonomy for Robust Robotic Manipulation under Deep Learning Uncertainty by Jongseok Lee 1 4 , Ribin Balachandran 1 , Harsimran Singh 1 , Jianxiang Feng 1 , Hrishik Mishra 1 , Marco De Stefano 1 , Rudolph Triebel 1 2 , Alin Albu-Schäffer 1 3 and Konstantin Kondak 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-05111.txt.
Introduction
Jongseok Lee and the rest of the team offer perceptive shared autonomy, where uncertainty figures from deep learning perception regulate autonomy level. When perception is confident, semi-autonomous handling raises scores; when uncertainty rises, control shifts to haptic teleoperation for toughness. An uncertainty-aware point cloud registration approach based on Neural Tangent Kernels enables the concept. Tested on hard aerial handling tasks in a 15-person user study and mock-up industrial cases, the SPIRIT system shows sound handling despite deep learning perception failures, raising scores and reliability.
Catalog record
| CREATOR | Jongseok Lee 1 4 , Ribin Balachandran 1 , Harsimran Singh 1 , Jianxiang Feng 1 , Hrishik Mishra 1 , Marco De Stefano 1 , Rudolph Triebel 1 2 , Alin Albu-Schäffer 1 3 and Konstantin Kondak 1 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85073 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.05111
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SPIRIT : Perceptive Shared Autonomy for Robust Robotic Manipulation under Deep Learning Uncertainty Jongseok Lee 1 4 , Ribin Balachandran 1 , Harsimran Singh 1 , Jianxiang Feng 1 , Hrishik Mishra 1 , Marco De Stefano 1 , Rudolph Triebel 1 2 , Alin Albu-Schäffer 1 3 and Konstantin Kondak 1 Abstract Deep learning (DL) has enabled impressive advances in robotic perception, yet its limited robustness and lack of interpretability hinder reliable deployment in safety-critical applications. We propose a concept termed perceptive shared autonomy, in which uncertainty estimates from DL-based perception are used to regulate the level of autonomy. Specifically, when the robot’s perception is confident, semi-autonomous manipulation is enabled to improve performance; when uncertainty increases, control transitions to haptic teleoperation for maintaining robustness. In this way, high-performing but uninterpretable DL methods can be integrated safely into robotic systems. A key technical enabler is an uncertainty-aware DL-based point cloud registration approach based on the so-called Neural Tangent Kernels (NTK). We evaluate perceptive shared autonomy on challenging aerial manipulation tasks through a user study of 15 participants and realization of mock-up industrial scenarios, demonstrating reliable robotic manipulation despite failures in DL-based perception. The resulting system, named SPIRIT, improves both manipulation performance and system reliability. SPIRIT was selected as a finalist of a major industrial innovation award. I Introduction In the fall of 1998, the interactive tour guide robot Minerva [ 43 ] was successfully deployed in the Smithsonian Museum. During its two weeks of operation, the robot traversed over 44 km while interacting with thousands of visitors. The key technical enabler was Minerva’s probabilistic algorithms. For perception, Minerva relied on probabilistic localization that explicitly estimated uncertainty in the robot’s state [ 13 ] . This information was then used to generate actions such that the robot could accomplish its mission despite uncertainty in sensing and motion [ 37 ] . In essence, Minerva showed how probabilistic approaches can enable robust operation in complex environments, where methods that ignored uncertainty failed to scale. Inspired by systems such as Minerva, probabilistic robotics led to subsequent progress over decades [ 9 , 35 , 2 ] . In contrast to the probabilistic foundations that enabled early deployments, modern robotics increasingly relies on deep learning (DL) for perception and decision-making [ 6 ] . Indeed, DL methods have enabled remarkable advances. Yet, their internal mechanisms remain difficult to interpret and their predictions can fail unexpectedly [ 42 , 16 ] , especially when test conditions are under-presented in training data [ 11 , 40 , 23 ] . As a result, integrating DL methods into high-stakes robotic applications remains challenging, not because such models are ineffective, but because their uncertainty is rarely accounted for at the system level. In this paper, we demonstrate that uncertainty-aware system design, rather than model perfection, is what enables the robust operation of robots – a lesson learned once in robotics through Minerva and revisited here for DL. Table I : The main novelty of the proposed system within the current state-of-the-art in probabilistic shared autonomy. Perception uncertainty Equipped with Equipped with Validation on from Deep Learning haptic feedback 3D visual feedback floating-base systems Learning human intent (i.e., [ 17 , 47 , 46 , 50 ] ) ✗ ✓ ✗ ✗ Adaptive authority allocation (i.e., [ 32 , 4 , 38 ] ) ✗ ✓ ✗ ✗ Probabilistic virtual fixtures (i.e., [ 29 , 1 , 34 ] ) ✗ ✓ ✗ ✗ Interactive and imitation learning (i.e., [ 18 , 50 ] ) ✗ ✗ ✗ ✗ The proposed system: SPIRIT ✓ ✓ ✓ ✓ This paper proposes a concept called perceptive shared autonomy for robust robotic manipulation under DL uncertainty (see Fig. ). With our concept, even if the used DL methods provide erroneous predictions when perceiving the environments, the robot is still able to accomplish the given tasks. We achieve this by allowing the robot’s users to actively transition between semi-autonomous manipulation and haptic teleoperation (“shared autonomy") [ 41 ] . Concretely, we model the uncertainty of the robot’s DL-based perception [ 25 ] . If the uncertainties are low, we transition to semi-autonomous manipulation for better performance. When the uncertainties are high, the user can transition to haptic teleoperation, opting for robustness. Here, special emphasis is placed on designing interactive features for the users. Thus, a human-robot interface is designed to intuitively provide the users with haptic and visual feedback about the robot’s current notion of perceptual uncertainty. We evaluate the proposed concept on aerial manipulation tasks, where robots manipulate objects while airborne, making this domain a suitable testbed for safety-critical applications due to the high operational risk. The developed system, named SPIRIT, is the main focus of this paper. In particular, our technical development is centered on SPIRIT’s DL-based perception, as uncertainty quantification in DL remains an open research challenge [ 15 ] . To address this, we propose a partitioned approach to point cloud registration, in which a digital twin of the environments is divided into local regimes. Within each regime, DL models are trained to perform prediction. Uncertainty is quantified using a Gaussian Process (GP) with a DL-based kernel called the Neural Tangent Kernel (NTK) [ 25 ] . Empirically, we first provide ablation studies to motivate the design of SPIRIT’s perception system. Robustness gains and overall system performance are evaluated within a user study with 15 participants and demonstration of industrial scenarios. SPIRIT’s prototype was deployed for a major industrial exhibition performing aerial manipulation demonstrations in public over five consecutive days. Later, SPIRIT was selected as a finalist of an innovation award. Contributions and major claims. Our main contribution is the development of SPIRIT, demonstrating robust robotic manipulation under DL uncertainty. We coin the term perceptive shared autonomy because we modulate the level of autonomy based on uncertainties in DL-based perception (Section III ). Our human-robot interface with haptic and 3D visual feedback that communicates the robot’s uncertainty is another novelty of this paper (Section III-B ). To realize SPIRIT, we propose a partitioned approach that learns to register point clouds with GP-based uncertainty estimation (Section IV ). We claim that our partitioned approach and GP-based uncertainty estimation enable SPIRIT’s development (Section V-A ). SPIRIT also empirically improves the robustness of the overall system by achieving the robotic manipulation tasks, even when DL-based perception has unexpectedly failed (Section V-B and V-C ). The video accompanying this paper illustrates SPIRIT’s system components and its behavior under DL uncertainty while performing challenging aerial manipulation tasks. Our project website can be found at https://sites.google.com/view/robotspirit . SPIRIT was originally developed for an industrial exhibition. The website disseminates those efforts. II Related Work Figure 1 : Amongst many potential applications, we examine two use cases: (a) extending the mobility of an inspection robot via pick and place, and (b) performing manipulation of industrial flange valves. The demonstration of these use cases along with the task sequence (1-4) is illustrated. Left: overview of the tasks. Right: robot performing the tasks from the left image. Our contribution lies in shared autonomy for robotic manipulation, building on probabilistic perception with DL. Tab. I summarizes the novelty: (a) leveraging uncertainty in DL-based perception, and (b) enabling user interaction with