3D Vision-tactile Reconstruction from Infrared and Visible Images for Robotic Fine-grained Tactile Perception

Yuankai Lin, Xiaofan Lu, Jiahui Chen, Hua Yang · 2025

Read 3D Vision-tactile Reconstruction from Infrared and Visible Images for Robotic Fine-grained Tactile Perception by Yuankai Lin, Xiaofan Lu, Jiahui Chen, Hua Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15087.txt.

Introduction

Yuankai Lin and co-authors build a touch sensor for robot fingers in a 2025 paper. To feel like a human fingertip, a gripper's soft sensing skin must curve, not lie flat. Flat touch sensors that read light inside the skin fail on curves: light falls unevenly, rebuilds blur, and edge rules grow complex.

Their system, GelSplitter3D, adds an infrared camera channel through a prism to the usual color view. A neural network trained with computer model ground truth turns the images into surface maps. The goal is fine touch reading on a curved, human like fingertip, so robots can judge grip and texture by feel.

Catalog record

CREATORYuankai Lin, Xiaofan Lu, Jiahui Chen, Hua Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15087

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

3D Vision-tactile Reconstruction from Infrared and Visible Images for Robotic Fine-grained Tactile Perception Yuankai Lin, Xiaofan Lu, Jiahui Chen, Hua Yang Abstract To achieve human-like haptic perception in anthropomorphic grippers, the compliant sensing surfaces of vision tactile sensor (VTS) must evolve from conventional planar configurations to biomimetically curved topographies with continuous surface gradients. However, planar VTSs have challenges when extended to curved surfaces, including insufficient lighting of surfaces, blurring in reconstruction, and complex spatial boundary conditions for surface structures. With an end goal of constructing a human-like fingertip, our research (i) develops GelSplitter3D by expanding imaging channels with a prism and a near-infrared (NIR) camera, (ii) proposes a photometric stereo neural network with a CAD-based normal ground truth generation method to calibrate tactile geometry, and (iii) devises a normal integration method with boundary constraints of depth prior information to correcting the cumulative error of surface integrals. We demonstrate better tactile sensing performance, a 40 % \% improvement in normal estimation accuracy, and the benefits of sensor shapes in grasping and manipulation tasks. I Introduction Humans have shown remarkable adaptability to grasping and manipulating tasks, ranging from grasping tools under various conditions to fine manipulation of tiny objects [ 1 ] . This capability, specifically the high-precision perception of tactile geometry, has fueled significant advances in robotic tactile sensing, which is now recognized as a pivotal enhancement for robotic perception systems [ 2 , 3 ] . The applications of these span a broad spectrum, including object detection [ 4 ] , six-degree-of-freedom (6-DOF) force estimation [ 5 ] , robust grasping mechanisms [ 6 ] , gait planning strategies [ 7 ] , and imitation learning frameworks [ 8 ] . These diverse applications necessitate the utilization of actuator ends with intricate surface geometries, equipped with highly accurate tactile sensors. Among all tactile sensors, the vision tactile sensor (VTS) has gained increasing adoption in robotic manipulation, due to its unique advantages, including high spatial resolution and multimodal sensing capabilities [ 9 ] . Despite these advancements, current VTSs predominantly adopt planar or simplistic curved configurations [ 10 , 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 ] , which fundamentally limit their compatibility with anthropomorphic designs. Such geometric constraints create two critical barriers: (1) reduced contact area continuity when interacting with complex object geometries, leading to incomplete tactile feedback; (2) inherent photometric distortion in curved sensor surfaces that undermines existing photometric stereo algorithms calibrated for planar geometries. Recent attempts to address these limitations through multi-camera arrays [ 20 ] or depth sensor fusion [ 21 ] have introduced new challenges in system miniaturization and computational complexity. Fig. 1 : GelSplitter3D sensors are deployed on a 12-Dof 3-finger gripper for robotic tactile fine-grained perception. Full-frame images is reconstructed into dense 3D point clouds while preserving pixel-level geometric details. To bridge this gap, we present GelSplitter3D – a curved VTS system that provides biomimetic geometry with computational perception, as shown in Fig. 1 . Unlike planar systems, curved surfaces exhibit large gradient variations. It is complex to achieve satisfactory imaging with three channels. The missing illumination information results in ambiguity and blurring in the reconstruction. Therefore, we propose RGB-NIR imaging to obtain more modal observations. Furthermore, a photometric stereo neural network (PSNN) is employed to estimate surface normals with a CAD-driven calibration process. Finally, to ensure geometric consistency, we incorporated boundary constraints based on depth priors. Our framework achieves 3D reconstruction of surfaces without the need for mathematical analysis. To improve the performance in curved tactile sensing, our contributions are threefold: 1. The optical imaging bottleneck is addressed through prism-mediated multichannel light-field redirection captured by an additional near-infrared (NIR) camera. 2. An PSNN architecture is established using RGB-NIR input and trained with CAD-simulated photometric patterns, enabling accurate surface normal estimation with a 40 \% decrease in MAE error. 3. A normal integration framework is developed that incorporates boundary constraints derived from prior sensor geometry, effectively reducing the propagation of cumulative errors in surface reconstruction. Qualitative and quantitative experiments validate the effectiveness of our GelSplitter3D sensor and the proposed method in the reconstruction of the curve surface. I-A Related Work Curved Surface VTSs: VTSs excel in high-resolution multimodal fusion for robotic manipulation. Recent advances in curved-surface VTS designs have demonstrated significant advantages over planar configurations, particularly in expanding sensing coverage and enhancing anthropomorphic compatibility. Romero et al . [ 10 ] pioneered light piping architectures that allow directional illumination control on curved surfaces. Liu et al . [ 20 ] engineered fingertip-shaped sensors with two cameras. Zhao et al . [ 22 ] introduced flexible backbone deformation models for torque estimation. Tippur et al . [ 14 ] further optimized compatibility with varied curved geometries through cross-LED illumination. GelStereo Palm [ 23 ] proposes a curved visuotactile sensor, which senses the 3D contact geometry using a binocular vision system. Gelstereo BioTip [ 24 ] presents a soft, biomimetic fingertip-sized visuotactile sensor, which can perceive 3D surface deformation. However, most of the curved surface vision-tactile sensors mentioned above have simple geometric shapes such as spheres or cones, which limits their sensing capability in fine manipulation tasks and generalization across different operational tasks. In this research, we propose a human finger-shaped curved surface VTS, capable of sensing the geometry of contact objects over a wide range, even in the absence of a mathematical analytical expression for the sensor’s surface, and able to replicate human actions such as pinching, grabbing, and gripping. Photometric Stereo Methods: In previous research, most VTSs utilizing photometric stereo for 3D reconstruction were limited to planar contact scenarios. To achieve 3D reconstruction on curved contact surfaces, researchers have explored various approaches. Zhang et al . [ 15 ] proposed a self-supervised deep learning-based photometric stereo method that obtains surface normals from single images without pre-calibration. Tippur et al . [ 16 ] developed a rainbow illumination scheme using semi-specular coatings to generate mixed color gradients. Gomes et al . [ 13 ] derived a projection function for curved sensors to map image-space pixels to physical surface points. Sun et al . [ 25 ] integrated photometric stereo with structured light, employing collimators to detect 3D deformations in flexible shells. However, existing photometric stereo methods primarily rely on RGB triple-channel information, demonstrating limited performance in fine texture perception. To address this limitation, we propose an RGB-NIR fusion photometric stereo neural network that significantly improves 3D reconstruction accuracy. Normal Integration: Current vision tactile 3D reconstruction predominantly employs a fast Poisson solver for depth recovery via normal integration, yet faces challenges of unknown boundary conditions and cumulative errors in curved surface reconstruction. Cao et al . [ 26 ] introduced a bilateral weighted functional for semi-smooth surfaces, enabling discontinuity-preserving normal integration under perspective projection. Xiu et al . [ 27 ] addressed global depth drift through variational optimization that reconstructs high-frequency details while incorporating low-frequency depth priors. Taketomi et al . [ 28 ] employed volume rendering to optimize a neural signed distance function (SDF), ensuring consistency between rendered and input normal maps. Building on these advances, we propose a normal integration framework that integrates boundary constraints with depth priors, effectively eliminating cumulative errors in curved

Read the complete text