LightTact: A Visual–Tactile Fingertip Sensor for Deformation-Independent Contact Sensing

Changyi Lin, Boda Huo, Mingyang Yu, Emily Ruppel, Bingqing Chen, Jonathan Francis, Ding Zhao · 2025

Read LightTact: A Visual–Tactile Fingertip Sensor for Deformation-Independent Contact Sensing by Changyi Lin, Boda Huo, Mingyang Yu, Emily Ruppel, Bingqing Chen, Jonathan Francis, Ding Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20591.txt.

Introduction

Changyi Lin and co-authors built a fingertip that sees touch without anything bending. Most tactile sensors infer contact from deformation, so contact with liquids, gels, or ultra-soft materials registers weakly or not at all. LightTact makes contact directly visible instead.

Its optical design blocks ambient light and internal illumination everywhere except true contact points, where scattered light passes through. Raw images are therefore near-black away from contact and show the contacting surface's natural appearance at contact. Pixel-level segmentation stays accurate across materials, forces, appearances, and lighting. Demonstrations include spreading water, dipping into facial cream, and handling soft thin films. The aligned visual-tactile images can even be read directly by vision-language models.

Catalog record

CREATORChangyi Lin, Boda Huo, Mingyang Yu, Emily Ruppel, Bingqing Chen, Jonathan Francis, Ding Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-20591

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LightTact: A Visual–Tactile Fingertip Sensor for Deformation-Independent Contact Sensing Thanks: Equal contribution Thanks: Carnegie Mellon University Thanks: Bosch Center for Artificial Intelligence (BCAI) Changyi Lin, Boda Huo, Mingyang Yu, Emily Ruppel, Bingqing Chen, Jonathan Francis, Ding Zhao Abstract Contact often occurs without macroscopic surface deformation, such as during interaction with liquids, semi-liquids, or ultra-soft materials. Most existing tactile sensors rely on deformation to infer contact, making such light-contact interactions difficult to perceive robustly. To address this, we present LightTact, a visual-tactile fingertip sensor that makes contact directly visible via a deformation-independent, optics-based principle. LightTact uses an ambient-blocking optical configuration that suppresses both external light and internal illumination at non-contact regions, while transmitting only the diffuse light generated at true contacts. As a result, LightTact produces high-contrast raw images in which non-contact pixels remain near-black (mean gray value < 3 <3 ) and contact pixels preserve the natural appearance of the contacting surface. Built on this, LightTact achieves accurate pixel-level contact segmentation that is robust to material properties, contact force, surface appearance, and environmental lighting. We further integrate LightTact on a robotic arm and demonstrate manipulation behaviors driven by extremely light contact, including water spreading, facial-cream dipping, and thin-film interaction. Finally, we show that LightTact’s spatially aligned visual-tactile images can be directly interpreted by existing vision-language models, enabling resistor value reasoning for robotic sorting. I Introduction Contact occurs throughout physical interactions between robots and their environments, for example, between fingertips and objects during manipulation or between feet and terrain during locomotion. To perform such interactions safely, precisely, and robustly, robots must be able to detect and localize contact [ 5 , 20 ] . Vision-based tactile sensors (VBTSs) have gained prominence in recent years due to their high spatial resolution and low cost [ 15 , 16 , 32 , 11 ] . However, most VBTSs sense contact indirectly by observing macroscopic deformation of a soft surface [ 35 , 33 , 18 , 30 , 36 ] . This deformation-dependent mechanism fundamentally requires measurable indentation to generate informative tactile signal. In contrast, many real-world interactions involve light contacts that induce little or no surface deformation. Examples include interactions with liquids or semi-liquids, and gentle contacts with extremely soft or thin materials. Such scenarios are common in everyday tasks, such as spreading water on a tabletop, dipping facial cream, or detecting a sheet of food film, but they remain challenging for deformation-based tactile sensors. Several sensors attempt to bypass deformation by using frustrated total internal reflection (TIR) to highlight the contact regions of a transparent medium [ 10 , 26 , 38 , 27 ] . However, ambient illumination and reflections from non-contact regions can also enter the camera, seriously corrupting the highlighted contact signal. As a result, TIR-based sensors typically function reliably only with monochromatic objects or under dark, tightly controlled environments–conditions incompatible with robotic tactile perception in everyday, unstructured settings. Thus, building a tactile sensor that can robustly detect contact without relying on surface deformation remains an open challenge. Figure 1: LightTact provides direct, pixel-level contact sensing across liquids, ultra-soft materials, and rigid objects, without requiring a minimum contact force. Its optical design produces high-contrast raw images in which non-contact pixels remain near-black, while contact pixels preserve the natural appearance of the contacting surface. LightTact is fingertip-sized and can be integrated into dexterous hands such as Amazing Hand [ 22 ] . To address this challenge, we introduce LightTact, a compact fingertip visual–tactile sensor that achieves direct, optics-based contact detection independent of surface deformation (Fig. 1 ). The key innovation is our dedicated optical configuration that entirely suppresses both external light and internal illumination at non-contact regions, while allowing only the diffuse light generated at true contacts to reach the camera. This design fundamentally changes how contact is encoded in the raw sensor image, enabling three core properties: • High-contrast raw images. LightTact captures raw images with naturally high-contrast contact representation: non-contact pixels remain near-black (intensity <2 ), while contacting regions appear relatively bright. • Robust pixel-level contact segmentation. Supported by the inherent optical isolation, for the first time, LightTact achieves pixel-level contact segmentation that is robust to material properties, contact forces, surface appearance, and environmental lighting. • Aligned visual–tactile multimodality. The bright pixels simultaneously encode the localized visual appearance of the object and the contact region of the sensor surface, producing a naturally aligned multimodal signal suitable for robot learning and high-level reasoning. Leveraging these properties, we integrate LightTact into a robotic system and demonstrate manipulation capabilities that remain difficult or inaccessible for existing tactile sensors. These include reliable interaction with liquids, semi-liquids, and extremely soft materials, as well as fine-grained sorting of tiny objects by directly prompting LightTact images to vision–language models. Together, these results indicate that LightTact enables a new class of manipulation behaviors by providing direct, optics-based sensing of true physical contact. In summary, our contributions are as follows: • We propose an ambient-blocking optical layout that theoretically isolates all external and internal illumination from non-contact regions, enabling intuitive contact visualization without relying on deformation. • We introduce LightTact, a compact fingertip visual-tactile sensor that achieves robust contact segmentation with minimal computation cost. • We systematically evaluate LightTact across diverse materials, contact areas, and lighting conditions, and report its success and failure cases. • We demonstrate that LightTact unlocks manipulation capabilities that require reliable detection of light contact, and supports multimodal reasoning by directly prompting LightTact images to vision–language models. II Related Work We review prior work in two aspects most relevant to this paper: (1) robust contact sensing and segmentation in vision-based tactile sensors (VBTSs), and (2) simultaneous visual–tactile sensing for robotic manipulation. II-A Contact Sensing with Vision-Based Tactile Sensors Vision-based tactile sensors (VBTSs) use internal cameras to capture visualized tactile signals, offering high spatial sensitivity at low cost. However, most existing VBTSs detect contact indirectly by observing macroscopic deformation of a soft sensing surface. Common approaches include tracking painted markers [ 33 , 30 , 1 , 4 , 36 , 37 ] , observing magnified reflective patterns [ 24 , 25 ] , or measuring luminance variations of reflective layers [ 35 , 18 ] . Fundamentally, these approaches require visible surface deformation, which imposes three constraints: the surface must be soft, the contacting object must be relatively harder, and the applied force must exceed a detectable deformation threshold. These requirements not only fix the sensor to a soft surface, but also severely limit its ability to perceive light contacts that do not generate measurable indentation, such as touching water, facial cream, or thin films. To decouple sensing from mechanical deformation, several sensors [ 10 , 26 , 38 , 27 ] apply total internal reflection (TIR) within a transparent medium. When an object contacts the medium, local frustration of TIR occurs, reflecting additional light toward the camera and thereby highlighting contact regions. However, this highlighting strategy is highly sensitive to ambient light. External light, reflections from non-contact portions of the object, and internal LED illumination can all enter the camera through non-contact regions, interfering with or overwhelming the intended TIR-based contact signal. As a result, these systems operate reliably only with monochromatic objects or under controlled, dark environments. In contrast, LightTact

Read the complete text