Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch
Gabriele M. Caddeo, Pasquale Marra, Lorenzo Natake · 2026
Read Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch by Gabriele M. Caddeo, Pasquale Marra, Lorenzo Natake (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-09100.txt.
Introduction
Caddeo, Marra, and Natale propose a physics-based method for object rebuild and pose finding under hand cover. Prior cover-aware methods use vision alone, but touch signals cut doubt. Body sense gives the posed hand shape, and touch constrains where the surface must lie. The method models object shape as a pose-aware signed distance field. Virtual tests show body sense and touch greatly improve filling under cover at true scale over vision-only rivals. Use on a real human-like robot with a new hand confirms transfer.
Catalog record
| CREATOR | Gabriele M. Caddeo, Pasquale Marra, Lorenzo Natake |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49346 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.09100
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch Gabriele M. Caddeo, Pasquale Marra, Lorenzo Natake Abstract We propose a multimodal, physically grounded approach for metric-scale amodal object reconstruction and pose estimation under severe hand occlusion. Unlike prior occlusion-aware 3D generation methods that rely only on vision, we leverage physical interaction signals: proprioception provides the posed hand geometry, and multi-contact touch constrains where the object surface must lie, reducing ambiguity in occluded regions. We represent object structure as a pose-aware, camera-aligned signed distance field (SDF) and learn a compact latent space with a Structure-VAE. In this latent space, we train a conditional flow-matching diffusion model, pretraining on vision-only images and finetuning on occluded manipulation scenes while conditioning on visible RGB evidence, occluder/visibility masks, the hand latent representation, and tactile information. Crucially, we incorporate physics-based objectives and differentiable decoder-guidance during finetuning and inference to reduce hand–object interpenetration and to align the reconstructed surface with contact observations. Because our method produces a metric, physically consistent structure estimate, it integrates naturally into existing two-stage reconstruction pipelines, where a downstream module refines geometry and predicts appearance. Experiments in simulation show that adding proprioception and touch substantially improves completion under occlusion and yields physically plausible reconstructions at correct real-world scale compared to vision-only baselines; we further validate transfer by deploying the model on a real humanoid robot with an end-effector different from those used during training. Figure 1 : Inference pipeline . We present a method to generate physically plausible 3D shape reconstructions by fusing vision with contact. In particular, we consider the active information of contact coming from tactile sensors distributed on the hand, and the negative information (non-interpenetration) coming from the hand geometry. Apart from these information, the method requires the Egocentric RGB Image of the object along with the masks of the hand and the object. Then, a Flow Transformer conditioned on multiple senses generates a valid SDF in a 3D grid in the same geometric domain of the hand. A Structured Latents ( SLat ) pipeline can then be used to generate a refined textured mesh, already at metric-scale. 1 Introduction Accurate object geometry is important to guide robotic manipulation. This includes knowledge of object surface to compute stable contact, free space for collision avoidance, and object shape to plan subsequent actions. Yet the moments when geometry matters most are also when vision is least reliable. During grasping and in-hand manipulation, the hand severely occludes the object, and single-view RGB reconstruction becomes fundamentally underconstrained. Vision-only reconstructions may look plausible while being unusable for control, e.g., intersecting the hand, missing true contact regions, or drifting in metric scale. Manipulation naturally produces physical evidence . Contacts and known hand kinematics provides useful cues to resolve the ambiguities caused by occlusion. Tactile sensing has long been used to estimate object properties ( [ 2 ] , [ 35 ] ) and improve robustness of grasping ( [ 46 , 27 ] ) and in-hand manipulation ( [ 51 , 41 ] ), and recent advances in tactile hardware and learning-based models have made touch increasingly practical as a perception signal. Yet these cues are rarely integrated as geometric constraints for dense 3D reconstruction under occlusion. Critically, contact provides direct constraints on geometry exactly where vision is missing: observed touches indicate where the surface must be, while physical feasibility ( e.g . , non-interpenetration with the end-effector) constrains where the object cannot be. Despite their relevance, such interaction cues are still less commonly integrated as explicit geometric constraints in dense 3D reconstruction under severe occlusion. In parallel, recent diffusion and flow-based 3D generative models have improved single-view 3D inference by learning strong shape priors, but they are typically optimized for visual plausibility and are rarely grounded in physical constraints [ 30 ] . Under heavy occlusion, this can produce samples that violate contact feasibility or drift in scale, limiting their impact for downstream robotic tasks ( e.g . , insertion [ 57 ] or pose estimation [ 65 ] ). We address this gap by treating in-hand 3D reconstruction as a physically grounded generative inference problem. Building upon [ 68 ] , we propose a multimodal approach for metric-scale amodal object reconstruction under severe hand occlusion from a single egocentric RGB observation, proprioceptive hand pose, and multi-contact tactile feedback. Our key contribution operates at the structure generation stage: we represent object geometry as a pose-aware, camera-aligned signed distance field (SDF), providing a continuous and differentiable representation that enables gradient-based physical constraints. We learn a compact latent space for SDF grids with a Structure-VAE and train a conditional flow-matching model to infer this latent. The model is pretrained on vision-only object images to learn a strong shape prior, then finetuned on occluded manipulation scenes while conditioning on visible RGB evidence and masks, a latent encoding of the posed hand, and a learned volumetric touch representation. To enforce physical validity, we incorporate two interaction-derived objectives: a contact-consistency loss that pulls the reconstructed surface toward observed contacts, and a non-interpenetration loss that pushes the object out of the hand interior. We further use these same terms as decoder-based physics guidance during sampling, steering generation toward shapes that satisfy contact and feasibility constraints. Since we reconstruct the object in the same 3D domain as the end-effector, the pose is implicitly estimated. Moreover, our method integrates naturally into state-of-the-art shape–appearance pipelines: the physically consistent structure can be passed to a downstream refinement/texturing module (made pose- and occlusion-aware) to obtain textured reconstructions consistent with the camera view and the occluding hand. Alternatively, our structure estimate can serve as input to other recent reconstruction frameworks ( e.g . , [ 58 ] ). Fig. 1 shows the inference pipeline of our approach. Across simulation experiments, we show that integrating proprioception and touch substantially improves amodal completion under severe occlusion, producing reconstructions that are both visually plausible and physically consistent in real-world scale compared to vision-only baselines. Finally, we validate real-world transfer by deploying the model on a humanoid robot with an end-effector different from those used during training, demonstrating real-world applicability and generalization across embodiments. In summary, our contributions are: • A method to inject physical constraints into a generative reconstruction pipeline. • A pose-aware, camera-aligned SDF representation and multimodal conditioning that align 2D evidence, hand pose, and tactile contacts in a shared 3D grid. • Simulation and real-robot results demonstrating improved amodal completion and cross-embodiment transfer relative to vision-only baselines. 2 Related Works 2.1 3D Generative Models Early 3D generative approaches used GANs [ 22 ] to synthesize various 3D representations such as point clouds [ 25 , 31 ] , NeRF-style radiance fields [ 5 , 4 , 40 , 45 ] , or SDF [ 20 ] from visual inputs, but often struggled to generalize beyond the training distribution. With the emergence of Diffusion Models [ 52 , 24 ] , subsequent works explored representations such as point clouds [ 36 , 37 , 67 ] , voxel grids [ 38 , 29 ] , Triplanes [ 6 , 48 , 72 ] , or Gaussians mixtures [ 73 ] , improving fidelity but frequently at high computational cost. To address efficiency, several recent methods