Consensus-Driven Uncertainty for Robotic Grasping based on RGB Perception
Eric C. Joyce, Qianwen Zhao, Nathaniel Burgdorfer, Long Wang, Philippos Mordohai · 2025
Read Consensus-Driven Uncertainty for Robotic Grasping based on RGB Perception by Eric C. Joyce, Qianwen Zhao, Nathaniel Burgdorfer, Long Wang, Philippos Mordohai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-20045.txt.
Introduction
A robot that misjudges an object's pose will fumble its grasp. Worse, pose estimators tend to sound sure even when they are wrong. Eric Joyce and co-authors teach a robot to doubt itself productively. Their lightweight networks predict, before any grasp is attempted, whether a grasp guided by a camera-based pose estimate will succeed. The robot can then hold off when the odds look bad.
Training data came from real images run through pose estimation, plus simulated grasping trials. Objects varied widely across trials. Surprisingly, networks trained on all objects together did better than specialists. Grasp success seems to share learnable cues across very different items. The work links two fields that rarely meet: pose uncertainty and the downstream act of grasping.
Catalog record
| CREATOR | Eric C. Joyce, Qianwen Zhao, Nathaniel Burgdorfer, Long Wang, Philippos Mordohai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-20045
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Consensus-Driven Uncertainty for Robotic Grasping based on RGB Perception Eric C. Joyce, Qianwen Zhao, Nathaniel Burgdorfer, Long Wang, Philippos Mordohai Abstract Deep object pose estimators are notoriously overconfident. A grasping agent that both estimates the 6-DoF pose of a target object and predicts the uncertainty of its own estimate could avoid task failure by choosing not to act under high uncertainty. Even though object pose estimation improves and uncertainty quantification research continues to make strides, few studies have connected them to the downstream task of robotic grasping. We propose a method for training lightweight, deep networks to predict whether a grasp guided by an image-based pose estimate will succeed before that grasp is attempted. We generate training data for our networks via object pose estimation on real images and simulated grasping. We also find that, despite high object variability in grasping trials, networks benefit from training on all objects jointly, suggesting that a diverse variety of objects can nevertheless contribute to the same goal. Data, code, and guides are hosted at: https://github.com/EricCJoyce/Consensus-Driven-Uncertainty/ I Introduction Remarkable progress in object pose estimation from single RGB images has been made in the past few years [ 1 , 2 , 3 , 4 ] , primarily driven by deep learning and the ability to reduce the so-called sim2real gap . This has enabled end-to-end system training on large amounts of synthetic data with precise ground truth. Despite these advances and comprehensive benchmarks such as the Benchmark for 6D Object Pose Estimation (BOP) [ 5 , 3 ] 1 1 1 https://bop.felk.cvut.cz/home/ , grasping on the basis of these estimates is still difficult, and the potential for deploying RGB-based pose estimators in downstream robotic applications remains unclear. Consider for example the pose estimates illustrated in Figure 1 . These were made by current methods, yet all four caused grasping attempts to fail when used as guides. Fig. 1: RGB-based pose estimates as green overlays and their corresponding ground-truth poses as solid objects. In our trials, grasping is attempted according to the estimated poses, and its success or failure is recorded as training data for an uncertainty-quantification network. All estimates seen here perform better than average according to established metrics and yet contain errors sufficiently large to cause grasping trials to fail. Motivated by this disconnect between pose evaluation and success in downstream grasping, we propose an approach to estimate the likelihood for success before a grasp is actually attempted. Our approach operates under the following assumptions: (i) perception occurs in RGB; (ii) objects are known and rigid; (iii) pose estimation occurs at the instance, not category, level; and (iv) a detector to locate the bounding box of the object in the image is available. These settings are common in the robotics literature [ 4 ] and match those of BOP. Pose estimate accuracy naturally improves with the inclusion of a depth channel [ 6 ] , but RGB-only sensing is ubiquitous and applicable in both indoor and outdoor settings. A grasping agent equipped with a pose estimator aware of its own uncertainty would be able to avoid harmful failures by refusing to attempt grasps based on dubious estimates. In their survey of under-studied challenges in monocular object pose estimation, Thalhammer et al. [ 4 ] urge researchers to focus on uncertainty in 6-DoF object pose estimation. One of the cited studies by Shi et al. [ 7 ] quantifies uncertainty but does not connect to the grasping task. In this paper, we propose a complete pipeline from perception, through uncertainty, to robotic grasping. This process begins by using off-the-shelf, high-performing 6-DoF pose estimators to make predictions about known objects in RGB images. As observed by the authors of FoundationPose [ 8 ] , obtaining and accurately annotating real-world data is prohibitive. We therefore perform grasping trials in a physics simulator using the collected pose estimates, and assemble a training set for multi-layer perceptrons (MLP). These networks then learn to predict grasp failures. The result is conceptually similar to an ensemble but differs in that one of the ensemble members is promoted to the role of Principal Estimator (PE). We therefore refer to our approach as a “consensus”, wherein the PE is responsible for the downstream grasp, and the differences between the principal and other members’ estimates becomes the basis for predicting the PE’s success. Our approach is compatible with any number of object detectors, pose estimators, and grippers. We compare our MLPs to a competitive baseline derived from Shi et al. and find that per-object grasp-failure prediction can be improved further by training one predictor jointly on all objects rather than individual, per-object predictors. In short, our contributions are: • the integration of visual perception on real imagery, a grasping simulator, and uncertainty quantification, providing a complete pipeline for grasping agents to make and qualify predictions, • a consensus-based type of predictor trained using off-the-shelf pose estimators and physics simulation results to assess a grasp attempt’s probability for success, • a grasping-trial dataset and means for collecting noise-free trial data. II Related Work In this section, we focus on approaches that share the key characteristics of ours: RGB-only perception for pose estimation at the instance level, given known objects with an image detector, and a grasp per gripper for each object. This puts adjacent research involving unknown objects [ 9 , 10 , 8 ] , category-level pose estimation [ 11 ] , depth sensors [ 12 , 11 , 8 ] , grasp learning [ 13 , 14 , 15 ] , gripper design [ 16 ] , and inverse kinematics and trajectory planning [ 17 ] out of scope. Note that, unlike our method, some of these methods (e.g. FoundationPose) require a depth channel. RGB Pose Estimation Methods: The progress made in pose estimation is evident in surveys [ 1 , 2 , 4 ] and on the BOP website. We categorize pose estimators into end-to-end (E2E) and PnP-based approaches. The former (e.g. [ 18 , 19 ] ) predict unmediated poses; the latter predict point correspondences which may be used to solve object poses using the Perspective- n -Points algorithm. E2E has been largely eclipsed by the PnP family. PnP-based methods produce a number of correspondences sufficient to compute an object’s pose. “Sparse” estimators [ 20 , 21 , 22 ] predict the projections of an object’s 3D bounding-box corners or 3D control points. Pose is then deduced from these minimal enclosures. When correspondences for almost every pixel are available, “dense methods” such as [ 23 , 24 , 25 , 26 , 27 , 28 , 29 ] gain robustness to outliers. A dense PnP-based method, EPOS [ 25 ] aims at robustness against textureless and symmetric objects by defining objects as sets of fragments. The network learns to predict probabilities for fragments to which a pixel might belong. Geometry-Guided Direct Regression (GDR-Net) [ 26 ] combines correspondence-based estimation and direct pose regression. GDR-Net generates dense 2D-3D correspondences as intermediate features before directly regressing pose using a learned, patch-based PnP approximator. Wang et al. credit their method’s success to thoughtful representations for rotation [ 30 ] and translation [ 31 ] , and to a loss function that combines pose