Heterogeneous Tactile Transformer

Jianxin Bi, Qiang Wang, Jayaram Reddy, Kelvin Lin, Soibkhon Khajikhanov, Ruihan Gao, Harold Soh · 2026

Read Heterogeneous Tactile Transformer by Jianxin Bi, Qiang Wang, Jayaram Reddy, Kelvin Lin, Soibkhon Khajikhanov, Ruihan Gao, Harold Soh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29948.txt.

Introduction

Bi and six colleagues tackle a portability problem in touch sensing. Tactile sensors differ so much that a model trained on one cannot be used on another, which blocks learning manipulation skills from mixed tactile data.

Their Heterogeneous Tactile Transformer gives each sensor its own encoder and shares a transformer trunk across sensors. Pretraining mixes masked reconstruction within each sensor type with alignment between paired sensors viewing the same contact. For this the authors built the Heterogeneous Paired Tactile dataset, 1.6 million synchronised paired frames from four vision-based and array-based tactile sensors. Across perception tasks and real manipulation, the learned representations transfer to new tasks and to sensors never seen in training.

Catalog record

CREATORJianxin Bi, Qiang Wang, Jayaram Reddy, Kelvin Lin, Soibkhon Khajikhanov, Ruihan Gao, Harold Soh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29948

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Heterogeneous Tactile Transformer Jianxin Bi, Qiang Wang, Jayaram Reddy, Kelvin Lin, Soibkhon Khajikhanov, Ruihan Gao, Harold Soh Abstract Tactile sensors are inherently heterogeneous: a model trained on one sensor cannot be directly used on another, which limits learning contact-rich manipulation policies from diverse tactile data at scale. To bridge this gap, we propose the Heterogeneous Tactile Transformer (HTT), a framework that learns shared tactile representations across heterogeneous sensors. HTT consists of sensor-specific encoders and a shared transformer trunk, and is pretrained with per-modality masked reconstruction together with cross-modal alignment between paired sensors. Pretraining uses our novel Heterogeneous Paired Tactile (HPT) dataset, containing 1.6M synchronized paired frames across four vision- and array-based tactile sensors. Across distinct tactile perception and real-world manipulation tasks, HTT is shown to learn transferable representations that adapt to new tasks and previously unseen sensors. Dataset, code, and model checkpoints will be released upon publication, at this url . Keywords: Tactile Representation, Contact-Rich Manipulation 1 Introduction Tactile sensing is critical for robot manipulation, especially in contact-rich tasks such as grasping fragile objects, precision assembly, and material identification. Yet tactile learning remains challenging to scale. One critical bottleneck is that, unlike cameras, tactile sensors are heterogeneous and differ in how they measure contact and in the data they produce. Optical tactile sensors such as GelSight-style devices [ 40 , 22 , 26 ] capture detailed spatial and geometric information from elastomer deformation. In contrast, array-based tactile sensors [ 33 , 1 , 32 , 17 ] provide high-rate temporal signals and more direct force or pressure measurements. These sensing modalities are complementary, but they are not easy to combine. Optical sensors typically provide rich spatial observations but are often limited by camera frame rates. Array-based sensors provide faster force-sensitive measurements, but with lower spatial resolution. More importantly, their raw outputs have very different structures: images in one case and time series over sensing elements in the other. This mismatch makes it difficult to train reusable tactile representations or sensor-agnostic manipulation policies. In this work, we seek a tactile backbone for heterogeneous sensors. Recent tactile representation learning has made strong progress, but it has largely focused on optical tactile sensors [ 14 , 43 , 12 , 5 , 36 ] . Self-supervised methods based on Masked Autoencoders (MAE) [ 13 ] can learn powerful features from GelSight-style images [ 43 , 12 ] . However, models trained only on optical tactile data inherit the limits of that sensor: they cannot directly ingest array-based tactile signals, and they may be less suitable for tasks that depend on high-rate force or slip cues. This motivates a heterogeneous tactile backbone that aligns sensor representations in a shared latent space, while preserving the structure and strengths of each sensor class. To this end, we introduce the Heterogeneous Tactile Transformer (HTT) , a self-supervised framework trained on our curated Heterogeneous Paired Tactile (HPT) dataset. HPT contains 1.6 M paired tactile frames from optical sensors (GelSight Mini, 9DTact) and array-based sensors (Xela, Tac-02), collected using a Universal Manipulation Interface (UMI) setup [ 3 , 38 ] . The dataset covers synchronized sensor-object interactions across press , twist , and slide motions. HTT uses sensor-specific encoders to process each sensor and a shared transformer trunk to form a common latent space. During pretraining, it combines masked reconstruction with bidirectional cross-sensor prediction. This allows the model to preserve sensor-specific structure while aligning representations across sensors. We evaluate HTT on three tactile perception tasks: object classification , force estimation , and slip detection . We further evaluate it on simulated and real-world robot manipulation tasks. Across these settings, HTT learns transferable representations for heterogeneous tactile sensors and improves downstream policy learning with novel tactile inputs. In particular, HTT-derived embeddings benefit contact-rich tasks such as toy screw and grasp tofu . These results suggest that paired heterogeneous pretraining is a practical path toward more general tactile backbones. In summary, our work studies how to learn unified tactile representations across heterogeneous sensors, with the goal of improving transfer to downstream tasks and novel sensor types. Our core contributions are as follows: • Heterogeneous Paired Tactile Dataset (HPT) , a large-scale tactile dataset containing 1.6 M paired frames from optical and array-based tactile sensors. HPT provides synchronized cross-sensor data for studying heterogeneous tactile representation learning. • Heterogeneous Tactile Transformer (HTT) , a self-supervised framework that combines MAE-style masked reconstruction with bidirectional cross-sensor prediction to learn shared representations across heterogeneous tactile sensors. • Comprehensive Evaluation across tactile perception tasks, simulated manipulation tasks, and real-world robot experiments. The results show that HTT learns transferable tactile representations and can serve as a general backbone for heterogeneous tactile sensing. Figure 1: Heterogeneous Tactile Transformer (HTT). HTT is pretrained on the Heterogeneous Paired Tactile (HPT) dataset , which consists of 1.6 M synchronized paired frames across four distinct sensors with a UMI device. HTT adopts sensor-specific encoders and a shared transformer trunk as core modules. Data from each sensor is patchified, fed into a sensor-specific encoder, and forwarded to the shared transformer trunk. During pretraining, decoders are used to reconstruct each sensors’ input data and cross-sensor predictors are used to predict masked target sensor embedding — aligning the shared latent space across heterogeneous sensors. The pretrained HTT model can be applied to distinct perception tasks and boost manipulation policy learning on unseen sensors. 2 Related Work This section gives a brief overview of tactile sensing, datasets, and representation learning for heterogeneous sensors; broader reviews of tactile sensing and tactile perception for robotics can be found in survey articles (e.g., [ 24 , 27 , 25 ] ). Tactile Sensors. Tactile sensors differ substantially in their transduction mechanisms and output formats. In this work, we focus on two broad families that are common in robot learning: optical tactile sensors and array-based tactile sensors. Optical tactile sensors, such as GelSight [ 40 , 10 ] , DIGIT [ 21 , 23 ] , 9DTact [ 26 ] , ThinTac [ 34 ] , and PolyTouch [ 42 ] , infer contact properties from images of a deformable contact surface. These sensors provide rich spatial information for texture, shape, and contact geometry, but are often limited by camera frame rates and sensor-to-sensor variation. Array-based tactile sensors, such as Xela uSkin [ 33 ] , PapillArray [ 20 ] , and TAC-02 [ 32 ] , measure contact through distributed sensing elements that convert physical stimuli into electrical signals. They provide high-rate force- or pressure-sensitive measurements but have lower spatial resolution than optical tactile sensors. These complementary properties motivate learning representations from both sensor families. Tactile Datasets for Robot Learning. Large-scale tactile representation learning depends on datasets that cover diverse sensors, objects, and interactions. Most public tactile datasets used for robot learning remain centered on optical tactile sensors [ 37 , 43 , 14 , 5 , 31 , 39 , 6 ] ; some provide paired observations across different optical sensors [ 5 ] , while others leverage physics-based simulation [ 12 , 30 ] or implicit neural representations [ 9 ] to increase scale and diversity. However, they largely remain within the optical tactile setting. To our knowledge, there is still no large-scale dataset of time- and space-synchronized

Read the complete text