Surformer v1: Transformer-Based Surface Classification Using Tactile and Vision Features

Manish Kansana 1 , Elias Hossain 1 , Shahram Rahimi 2 , Noorbakhsh Amiri Golilarz 2 · 2025

Read Surformer v1: Transformer-Based Surface Classification Using Tactile and Vision Features by Manish Kansana 1 , Elias Hossain 1 , Shahram Rahimi 2 , Noorbakhsh Amiri Golilarz 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06566.txt.

Introduction

Robots that touch a surface can learn what it is made of from both touch and sight. Manish Kansana, Elias Hossain, Shahram Rahimi, and Noorbakhsh Amiri Golilarz built a system for that task in 2025, named Surformer v1. It pairs touch features with compact visual features taken from an image network, and lets the two streams interact through attention.

A touch-only version of the model gave the best accuracy among the methods tried and ran fastest. Adding sight kept accuracy high, at 99.4 percent, with a response time under a millisecond. A rival image-based model scored slightly higher but took far longer, so the authors stress the balance of speed and accuracy.

Catalog record

CREATORManish Kansana 1 , Elias Hossain 1 , Shahram Rahimi 2 , Noorbakhsh Amiri Golilarz 2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-06566

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Surformer v1: Transformer-Based Surface Classification Using Tactile and Vision Features Manish Kansana 1 , Elias Hossain 1 , Shahram Rahimi 2 , Noorbakhsh Amiri Golilarz 2 Abstract Surface material recognition is a key component in robotic perception and physical interaction, particularly when leveraging both tactile and visual sensory inputs. In this work, we propose Surformer v1, a transformer-based architecture designed for surface classification using structured tactile features and Principal Component Analysis (PCA)-reduced visual embeddings extracted via ResNet 50. The model integrates modality-specific encoders with cross-modal attention layers, enabling rich interactions between vision and touch. Currently, state-of-the-art deep learning models for vision tasks have achieved remarkable performance. With this in mind, our first set of experiments focused exclusively on tactile-only surface classification. Using feature engineering, we trained and evaluated multiple machine learning models, assessing their accuracy and inference time. We then implemented an encoder-only Transformer model tailored for tactile features. This model not only achieves the highest accuracy, but also demonstrated significantly faster inference time compared to other evaluated models, highlighting its potential for real-time applications. To extend this investigation, we introduced a multimodal fusion setup by combining vision and tactile inputs. We trained both Surformer v1 (using structured features) and Multimodal CNN (using raw images) to examine the impact of feature-based versus image-based multimodal learning on classification accuracy and computational efficiency. The results showed that Surformer v1 achieved 99.4% accuracy with an inference time of 0.77 ms, while the Multimodal CNN achieved slightly higher accuracy but required significantly more inference time. These findings suggest that Surformer v1 offers a compelling balance between accuracy, efficiency, and computational cost for surface material recognition. The results also underscore the effectiveness of integrating feature learning, cross-modal attention and transformer-based fusion in capturing the complementary strengths of tactile and visual modalities. I Introduction Robots interacting with the physical world must be able to accurately perceive and classify the surfaces they touch. This capability is vital for safe object manipulation, navigation over varied terrains, and executing fine-grained tasks in unstructured environments. Tactile sensing provides detailed information about surface compliance, friction, and texture, properties that vision alone may struggle to capture reliably, especially under occlusions, poor lighting, or specular reflections. On the other hand, vision offers global context and appearance cues, making the combination of visual and tactile sensing a powerful solution for material recognition and robotic perception. Recent research has extensively explored both vision-based and tactile-based models. Vision-only models [ 1 , 2 ] , particularly those leveraging deep Convolutional Neural Networks (CNNs) and vision transformers, have achieved strong performance on image recognition tasks. However, their reliance on visual features makes them sensitive to occlusion, illumination variations, and specular noise. On the other hand, tactile-only models have demonstrated robustness in manipulation, object understanding, grasping and tactile data classification tasks where physical contact is critical [ 3 , 4 , 5 , 6 ] . For example, GelSight-based tactile sensors have enabled the extraction of high-resolution surface deformations, which are highly informative for material property inference. To harness the strengths of both sensing modalities, multimodal learning strategies have been introduced to improve surface and object understanding. Early efforts employed simple feature-level fusion (e.g., concatenation) or decision-level fusion techniques [ 7 , 8 ] , but these often failed to capture the nuanced relationships between heterogeneous modalities. More advanced fusion architectures have been proposed, including those using attention-based networks and cross-modal deep representations [ 9 , 10 , 11 ] . Models such as [ 12 , 13 , 14 ] have been developed for grasp planning, object exploration, and even navigation while interacting with aliased environments highlighting the complementary nature of tactile and visual data. In parallel, transformer-based models have emerged as a powerful backbone for multimodal representation learning due to their ability to model long-range dependencies and perform cross-modal reasoning. Architectures such as [ 15 , 16 ] have shown remarkable generalization in vision-language tasks, suggesting their potential for cross-sensory integration in robotics. Moreover, models like MAE [ 17 ] and CLIP [ 18 ] demonstrate the effectiveness of attention-based learning in fusing distributed representations from different input spaces. Despite these advances, several critical limitations persist in current surface classification approaches. First, end-to-end deep learning models typically rely on large volumes of labeled data, which are often scarce in tactile domains due to the complexity and cost of sensor setup and manual annotation. This data dependency not only limits scalability but also hampers generalizability in real-world scenarios. Second, many existing multimodal methods employ simplistic fusion strategies, such as feature concatenation or prediction averaging, which are inadequate for modeling the rich, nonlinear relationships and contextual dependencies between tactile and visual modalities. These approaches often ignore the dynamic interactions that naturally occur when humans perceive and interpret materials through both vision and touch. Third, most prior models lack modularity and transparency to incorporate both structured tactile information and learned visual embeddings and often treat tactile and visual inputs as homogeneous data streams without accounting for their distinct structures and complementary roles. As a result, there remains a need for models that are both effective and computationally efficient, particularly for resource-constrained robotic platforms. To address these limitations, we propose Surformer v1 , a transformer-based architecture designed specifically for surface classification. The model processes structured tactile features alongside reduced visual embeddings, combining them through a unified mid-level fusion framework. Unlike earlier methods, Surformer v1 enables cross-modal representation learning while maintaining computational efficiency, ensuring scalability to real-world robotic environments. Our key contributions are as follows: • We introduce a multimodal architecture, leveraging both tactile and PCA-reduced ResNet 50-based visual embeddings suitable for robotic deployment and extension to other sensor combinations. • We employ a mid-level fusion architecture with multi-head cross-attention layers to learn complementary representations, joint embeddings, and bidirectional interaction between tactile and visual inputs. • We implement a lightweight, encoder only Transformer for tactile-only classification, achieving the highest accuracy and the lowest latency, making it suitable for real-time deployment on embedded robotic systems where computational resources are limited. • We evaluate our models on the Touch and Go dataset [ 19 ] and demonstrate that multimodal fusion improves accuracy over tactile-only baselines. The rest of this paper is organized as follows: Section II details the Surformer v1 architecture. Section III describes the dataset and preprocessing pipeline, experimental results and baseline comparisons. Finally, Section IV concludes the paper and discusses future work. Fig. 1: Architecture of Surformer v1 for multi-modal surface classification from tactile and visual data. The model accepts two input modalities: (1) visual features extracted from surface images using a pretrained ResNet-50 model and reduced to 64 dimensions via PCA, and (2) the top 7 tactile features selected based on importance analysis. Each input passes through a modality-specific encoder to map it into a unified 128-dimensional latent space using dense layers, batch normalization, and dropout. In the Cross-Modal Fusion stage, both self-attention and bidirectional cross-attention mechanisms are applied to capture intra- and inter-modality relationships. Feed-forward networks refine features within each modality before they are concatenated and passed through a fusion block involving residual connections, projection layers, and normalization. The fused representation is finally processed by a classification head comprising three fully connected layers and a softmax output layer to

Read the complete text