ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation

Jayjun Lee Nima Fazeli · 2025

Read ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation by Jayjun Lee Nima Fazeli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-12239.txt.

Introduction

Jayjun Lee and Nima Fazeli wrote a 2025 robotics paper on sensing both where a held object sits and where it touches the world. Dexterous hands need both facts at once, yet sight and touch data are partial and noisy. Their system fuses camera views with rich fingertip touch signals. Objects are modeled as signed distance fields.

Touch is rendered as neural fields of shearing force across the surface. The system then pins contact spots onto the object's 3D shape. Training ran in simulation, with a bridge that carries the skill to real hardware untaught. Tests in both settings showed the method works in hard handling scenes.

Catalog record

CREATORJayjun Lee Nima Fazeli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-12239

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation Jayjun Lee Nima Fazeli Abstract Mastering dexterous, contact-rich object manipulation demands precise estimation of both in-hand object poses and external contact locations—tasks particularly challenging due to partial and noisy observations. We present ViTaSCOPE: Visuo-Tactile Simultaneous Contact and Object Pose Estimation, an object-centric neural implicit representation that fuses vision and high-resolution tactile feedback. By representing objects as signed distance fields and distributed tactile feedback as neural shear fields, ViTaSCOPE accurately localizes objects and registers extrinsic contacts onto their 3D geometry as contact fields. Our method enables seamless reasoning over complementary visuo-tactile cues by leveraging simulation for scalable training and zero-shot transfers to the real-world by bridging the sim-to-real gap. We evaluate our method through comprehensive simulated and real-world experiments, demonstrating its capabilities in dexterous manipulation scenarios. I Introduction Dexterous manipulation in contact-rich environments requires a seamless interplay of perception and control, particularly in unstructured settings where visual occlusions and unpredictable forces are the norm. From grasping a tool to executing precise interactions with the environment, the ability to reconstruct an object’s geometry, estimate its in-hand pose, and sense external contacts is critical for enabling effective manipulation. These capabilities are not merely enhancements; they are foundational for planning, predicting, and executing actions. Key to unlocking these capabilities is the ability to exploit the complementary nature of vision and touch. However, achieving this level of perception remains challenging due to the inherent ambiguity of partial observations from vision or tactile sensing alone. Vision offers a global but often occluded view of the environment, while tactile sensing provides high-resolution, localized contact information that complements vision but is limited to regions of intrinsic contact. Bridging these modalities is essential for integrating global context with fine-grained contact details, which requires understanding not only the 3D geometry of objects but also the behavior of extrinsic contacts that induce shear forces on tactile sensor membranes. To address these challenges, we propose ViTaSCOPE: a unified neural implicit representation for Visuo-Tactile Simultaneous extrinsic Contact and in-hand Object Pose Estimation. ViTaSCOPE introduces a novel framework that integrates vision and high-resolution tactile feedback to reconstruct object geometry, estimate in-hand object pose, and localize extrinsic contacts. Our approach offers the following key contributions: • We propose a contact-aware implicit representation that encodes object geometry as a signed distance field (SDF) and registers extrinsic contacts to the 3D geometry by conditioning on tactile shear field measured by vision-based tactile sensors. This enables seamless fusion of vision and tactile observations as an implicit function, enabling functionalities such as object geometry reconstruction, in-hand pose estimation, and extrinsic contact localization. Unlike traditional methods, our representation provides infinite resolution for object geometry and tactile shear fields, unlocking new capabilities for precision manipulation. • ViTaSCOPE offers a modularized approach to integrating distributed tactile feedback for contact estimation where its inference process is interpretable. • We scale up the dataset for extrinsic contact estimation using a penalty-based tactile model in GPU-accelerated simulation [ 1 ] . This approach enables robust sim-to-real transfer, leveraging the shear field tactile feedback for reliable performance compared to tactile RGB images. II Related Works II-A Simulation of Vision-based Tactile Sensors Vision-based tactile sensors such as DIGIT [ 2 ] , GelSight [ 3 ] , GelSlim [ 4 , 5 ] , Soft Bubble [ 6 ] , and DIGIT360 [ 7 ] provide high-resolution contact information through the observation of elastometric sensor membrane that deforms upon contact. The tactile sensors not only provide local texture and geometric information about the object from its contact patch, but also contain information about shear and normal force distributions during contact. These distributions describe pressure, shear, torsion, and slip along the tactile membrane. Recently, there has been advances in simulating these vision-based tactile sensors. TACTO [ 8 ] can render realistic high-resolution tactile signatures as RGB-D images by simulating rigid-body contacts between the elastomer and a rigid object. TacSL [ 1 ] adopts the soft contact penalty-based tactile model [ 9 ] with rigid bodies which allows interpenetration to simulate imaging process where contact force fields can be computed with GPU-accelerated Physics in Issac Gym [ 10 ] . Taxim [ 11 ] proposes an example-based method that combines optical simulation and marker motion field simulation that is calibrated with contact examples. DiffTactile [ 12 ] introduces a physics-based differentiable tactile simulation of contact force distribution and surface deformation using Finite-Element Method-based soft body model for the elastomers. M3L [ 13 ] uses global visual image with shear force field images in MuJoCo [ 14 ] with its touch-grid sensor plugin. In this work, we use TacSL to simulate shear force field observations from extrinsic contact events during prehensile manipulation with tools. II-B Implicit Object and Contact Representations Prior works on dense 3D geometry representations provide high-fidelity surface reconstructions for complex volumetric geometries. DeepSDF [ 15 ] introduces a learned continuous signed distance function (SDF) representation that enables high-quality shape representation, interpolation, and completion from partial and noisy 3D input data in canonical pose of the object. VIRDO [ 16 , 17 ] extends these concepts by integrating multimodal feedback from vision and wrench from force/torque sensing to learn rich latent embeddings of contact locations and forces to predict tool deformations subject to extrinsic contacts. Building on these foundations, NDCF [ 18 ] jointly models object deformations and extrinsic contact patches from similarly vision and wrench feedback using implicit representations, while ensuring that the predicted contacts lie on the object’s surface. NCF [ 19 ] tracks extrinsic contact by an implicit function that maps a query point to a contact probability, which does not consider multi-modal inputs, only vision-based tactile inputs and assumes rigid, known object models during inference. NISP [ 20 ] uses physics-informed neural networks to solve inverse source problems such as identifying unknown source functions for detecting intrinsic and extrinsic contacts via an implicit representation with physics integration. Despite these advancements, prior approaches [ 16 , 18 , 19 ] assume rigid grasps or extends the kinematic chain that directly associate the object pose with the end-effector pose. This simplification ensures that real-world point cloud observations align with the object’s canonical pose, removing the need for an additional explicit optimization of object pose in \mathrm{SE}(3) transformation space [ 15 ] before model inference. In contrast, our work removes the assumption of rigid grasps, allowing for in-hand object translations and rotations to address real-world observations in the world frame rather than the canonical pose. II-C Object Shape Reconstruction and Pose Estimation Accurate object pose estimation is essential for robotics applications such as manipulation, navigation, and human-robot interaction. Vision-based methods, coupled with multimodal sensing, have significantly advanced the precision and robustness of 6D pose estimation, particularly in dynamic and unstructured environments. Vision-based methods have significantly advanced object shape reconstruction and pose estimation, leveraging deep learning and implicit representations. AlignSDF [ 21 ] combines SDF with parametric models to reconstruct hands and objects from monocular RGB images, disentangling pose and shape for improved accuracy. FoundationPose [ 22 ] introduces a generalizable model for 6D pose estimation across diverse datasets, while What’s in Your Hands? [ 23 ] focuses on reconstructing hand-held objects from single RGB images using

Read the complete text