Towards Generalization of Tactile Image Generation: Reference-Free Evaluation in a Leakage-Free Setting

Cagri Gungor 1, Derek Eppinger 2, Adriana Kovashka 1, 2] · 2025

Read Towards Generalization of Tactile Image Generation: Reference-Free Evaluation in a Leakage-Free Setting by Cagri Gungor 1, Derek Eppinger 2, Adriana Kovashka 1, 2] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-06860.txt.

Introduction

Cagri Gungor, Derek Eppinger, and Adriana Kovashka tackled in 2025 a quiet flaw in how machines that fake touch images are graded. Touch data are scarce and costly, so generated images fill the gap in robotics and vision work. The team show that common datasets let training and test items overlap, inflating scores and hiding weak generalization. They propose a leak-free grading protocol with new scores that need no reference image.

Their own generator routes through text: short, material-specific descriptions guide it during training to catch fine contact traits. On two standard touch datasets, their method beat rivals under the strict grading. The authors argue the field should adopt leak-free tests so claimed progress in fake touch is real.

Catalog record

CREATORCagri Gungor 1, Derek Eppinger 2, Adriana Kovashka 1, 2]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-06860

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Generalization of Tactile Image Generation: Reference-Free Evaluation in a Leakage-Free Setting Cagri Gungor 1, Derek Eppinger 2, Adriana Kovashka 1, 2] Abstract Tactile sensing, which relies on direct physical contact, is critical for human perception and underpins applications in computer vision, robotics, and multimodal learning. Because tactile data is often scarce and costly to acquire, generating synthetic tactile images provides a scalable solution to augment real-world measurements. However, ensuring robust generalization in synthesizing tactile images—capturing subtle, material-specific contact features—remains challenging. We demonstrate that overlapping training and test samples in commonly used datasets inflate performance metrics, obscuring the true generalizability of tactile models. To address this, we propose a leakage-free evaluation protocol coupled with novel, reference-free metrics—TMMD, I-TMMD, CI-TMMD, and D-TMMD—tailored for tactile generation. Moreover, we propose a vision-to-touch generation method that leverages text as an intermediate modality by incorporating concise, material-specific descriptions during training to better capture essential tactile features. Experiments on two popular visuo-tactile datasets, Touch and Go and HCT, show that our approach achieves superior performance and enhanced generalization in a leakage-free setting. 1 Introduction The tactile modality, which relies on direct physical contact to sense attributes such as force, pressure, and temperature, is a unique sensory channel that is fundamental to our understanding of the world [ 19 , 27 ] . In computer vision and robotics, tactile cues effectively complement visual data in multimodal learning, enabling systems to discern subtle material properties that pure vision might overlook [ 7 , 49 , 6 , 23 ] . This integration not only provides critical feedback for robotic manipulation through nuanced contact dynamics [ 50 , 17 , 29 ] , but also enriches scene understanding by augmenting 3D representations with tactile insights [ 13 ] . However, obtaining robust and generalizable models remains a significant challenge across various tactile sensing tasks due to the expensive physical data collection process and non-standardized sensor outputs [ 47 ] . Figure 1 : An illustration of how our reference-free metrics evaluate generated tactile images across different classes. Within each class (e.g., “Concrete” or “Grass”), samples are split into two subsets (A and B) to measure intra-class consistency , while inter-class divergence quantifies how distinct the generated samples are between classes. Tactile generation stands out as a promising avenue for synthesizing tactile images from visual and multimodal inputs, offering a potential solution to mitigate the scarcity of tactile data. Recent research [ 46 , 13 , 38 , 32 , 16 ] in this area has demonstrated encouraging progress; however, the evaluation of these models remains problematic for two following reasons: data leakage and lack of specialized metrics. Data leakage poses a critical obstacle to advancing tactile sensing research, undermining the goal of robust generalization by compromising many prevalent tactile datasets [ 45 , 15 , 23 ] . These datasets, often derived from continuous video recordings of object interactions, inadvertently include overlapping or near-duplicate samples across training and test splits. This leakage enables models to simply memorize redundant tactile patterns. It also results in artificially inflated performance metrics across various tactile-related tasks that do not reflect real-world robustness. Consequently, current evaluations mask the true challenges of different tasks such as generating high-fidelity tactile images that capture the subtle, material-specific features. As a first contribution, we thoroughly analyze the leakage phenomenon—detailing its causes and impact—and introduce a leakage-free evaluation protocol that rigorously separates training and test subsets, thereby enabling more reliable and generalizable results, meaning consistent performance across diverse, unseen scenarios to address data variability. In evaluating tactile image generation, many works have adopted conventional metrics—such as SSIM, PSNR, LPIPS, and FID—that were originally designed for natural images. These metrics often fail to accurately capture tactile-specific nuances on generated tactile images. They exhibit excessive sensitivity to irrelevant background artifacts due to passive sensor state, in the absence of contact to a material. Meanwhile they neglect fine, material-specific features that are essential for tactile perception. This misrepresentation skews performance evaluations and obscures the true generalizability of the models. As a second contribution, we perform an in-depth analysis of the limitations of these metrics and we introduce a set of novel, reference-free evaluation metrics—namely, TMMD, I-TMMD, CI-TMMD, and D-TMMD—which are specifically designed to capture material-specific tactile features and eliminate the need for reference images. By leveraging a dedicated tactile encoder, these metrics quantify both intra-class consistency (ensuring internal stability) and inter-class divergence (ensuring diversity across materials) as illustrated in Fig. 1 , thereby offering an evaluation to accurately assess tactile image fidelity and diversity of generated images. This comprehensive evaluation ultimately fosters the development of more reliable and generalizable tactile generation systems. Finally, we propose a method that leverages text as an intermediate modality in the vision-to-touch generation task. This is achieved by incorporating concise, material-specific textual descriptions during training. By leveraging text to emphasize essential tactile features and filter out irrelevant visual details, our method accurately captures intrinsic contact patterns and material properties, thereby enhancing the quality of the generated tactile images. We use our robust evaluation framework, which operates under an enforced leakage-free setting and utilizes novel metrics tailored specifically for tactile generation, to verify the effectiveness of our method, ensuring that the results are reliable and generalizable. 2 Related Work Tactile sensing and visual-tactile datasets. In recent years, various tactile sensors have been developed and extensively utilized in robotics [ 52 , 4 , 50 , 5 , 14 ] . Traditional sensors [ 8 , 22 ] captured low-dimensional signals such as pressure, force, and vibration, providing limited surface details. To further enhance robotic interaction and manipulation, vision-based tactile sensors, such as GelSight [ 48 ] , DIGIT [ 25 ] , and Taxim [ 35 ] , overcome this by using elastomeric materials with embedded cameras and lighting for high-resolution tactile imaging. Many visual-tactile datasets, such as Touch and Go [ 45 ] , HCT [ 15 ] , SSVTP [ 23 ] , VisGel [ 26 ] and TaRF [ 13 ] have been developed using vision-based tactile sensors to support various computer vision tasks, including self-supervised representation learning [ 47 , 15 ] , material classification [ 45 , 47 ] , tactile-driven image stylization [ 47 , 15 ] , and cross-modal image generation [ 13 , 46 , 38 ] . However, our analysis reveals that some of these datasets include highly similar samples in both training and test sets, leading to overfitting and inflated performance metrics that poorly reflect real-world generalization. Cross-modal generation. Humans naturally connect information across different senses, a remarkable ability that has inspired extensive research in cross-modal generation. Early works laid the foundation by developing generative frameworks that could convert images from one representation to another [ 20 , 28 , 53 , 40 ] . More recently, diffusion‐based methods have become popular in cross‐modal image translation because they tend to produce high‐quality, photorealistic images with stable training dynamics [ 10 , 33 ] . These advances have been extended to a variety of conditioning modalities, enabling impressive results in tasks such as text‐to‐image generation [ 24 , 43 , 44 , 1 ] , audio‐to‐image synthesis [ 36 , 2 ] , and video‐to‐audio translation [ 30 ]

Read the complete text