ViTaPEs : Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformers

Fotios Lygerakis, Ozan Özdenizci, Elmar Rückert · 2025

Read ViTaPEs : Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformers by Fotios Lygerakis, Ozan Özdenizci, Elmar Rückert (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20032.txt.

Introduction

Fotios Lygerakis, Ozan Özdenizci, and Elmar Rückert present a 2025 paper on teaching machines to join sight and touch. Touch gives facts that sight misses, such as how soft or rough a thing feels. Past systems joined the two senses in weak ways and leaned on large pre-trained models.

Their system, called ViTaPEs, adds place codes in two steps. Codes go into each sense stream first, then a shared code goes in just before the model mixes the two. Tests on large real world sets show gains over past methods. The system also did well on new tasks and at guessing if a robot grip would hold.

Catalog record

CREATORFotios Lygerakis, Ozan Özdenizci, Elmar Rückert
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE69 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20032

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ViTaPEs : Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformers Fotios Lygerakis, Ozan Özdenizci, Elmar Rückert Abstract Tactile sensing provides local essential information that is complementary to visual perception, such as texture, compliance, and force. Despite recent advances in visuotactile representation learning, challenges remain in fusing these modalities and generalizing across tasks and environments without heavy reliance on pre-trained vision-language models. Moreover, existing methods do not study positional encodings, thereby overlooking the multi-scale spatial reasoning needed to capture fine-grained visuotactile correlations. We introduce ViTaPEs , a transformer-based framework that robustly integrates visual and tactile input data to learn task-agnostic representations for visuotactile perception. Our approach exploits a novel multi-scale positional encoding scheme to capture intra-modal structures, while simultaneously modeling cross-modal cues. Unlike prior work, we provide provable guarantees in visuotactile fusion, showing that our encodings are injective, rigid-motion-equivariant, and information-preserving, validating these properties empirically. Experiments on multiple large-scale real-world datasets show that ViTaPEs not only surpasses state-of-the-art baselines across various recognition tasks but also demonstrates zero-shot generalization to unseen, out-of-domain scenarios. We further demonstrate the transfer-learning strength of ViTaPEs in a robotic grasping task, where it outperforms state-of-the-art baselines in predicting grasp success. Project page: https://sites.google.com/view/vitapes 1 Introduction In nature, no form of intelligence exists without tactile perception. The sense of touch is fundamental to how humans and intelligent beings interact with the world. For humans, touch is critical for tasks such as grasping, manipulation, material characterization, and detecting environmental changes ( Lederman & Klatzky 1987 ; Klatzky & Lederman 1992 ) . It provides essential information about object properties like texture, compliance, and force, which is vital for fine motor skills and subtle interactions ( Calandra et al. 2017 ) . Tactile sensing can offer local descriptions of deformation at contact points, providing information that other modalities cannot efficiently capture. When combined with vision, it enhances perception by offering fine-grained details like pressure distributions and surface compliance, complementing vision’s global view of object shapes and spatial relationships ( Dahiya et al. 2010 ; Calandra et al. 2017 ) . Together, these modalities provide a thorough contextual understanding of the environment. Figure 1 : Task-accuracy radar comparing visuotactile models. ViTaPEs outperforms all others in robustness and cross-domain generalization. Recent works in visuotactile representation learning have shown the potential for joint models, improving performance in complex tasks that rely on both modalities ( Chen et al. 2022 ; Yang et al. 2024 ; Fu et al. 2024 ; Lygerakis et al. 2024 ) . Although a number of these methods can effectively learn shared representations, challenges remain, including aligning data across different sensory scales and handling domain-specific artifacts (e.g., texture, compliance, localization, scene context) and explicitly modeling positional encodings for multi-scale spatial alignment of touch with vision. Current research often relies on large, pre-trained visual or vision-language models (VLMs) ( Yang et al. 2024 ; Fu et al. 2024 ) , where the visual encoder is frozen and only the tactile encoder is trained to align with it. This can limit expressivity and assumes that visual representations are optimal for tactile alignment, hindering joint representation learning. A notable ViT-based exception is VTT ( Chen et al. 2022 ) , but it is trained solely on simulated force-torque feedback, far lower in resolution and complexity than data from high-resolution spatial sensors like DIGIT ( Lambeta et al. 2020 ) , GelSight ( Yuan et al. 2017a ) , or uSkin ( Tomo et al. 2018 ) . Moreover, VTT depends on application-specific auxiliary losses, and its generalization to broader tasks remains untested. Another key limitation is the narrow scope of most existing approaches. Current models are typically fine-tuned for specific downstream tasks, such as object manipulation, material classification, texture recognition, or cross-modal generation, reducing their ability to generalize ( Dave et al. 2024 ; Jang et al. 2022 ; Fang et al. 2024 ; Yang et al. 2023 ) . As a result, they often lack the versatility needed for broader applications. A more task-agnostic approach that performs well with little or no fine-tuning would significantly improve the practical utility of visuotactile representation learning. We present a cross-modal method that integrates visual and tactile data using a transformer-based architecture enriched with multi-scale Vi suo- Ta ctile P ositional E ncoding s , namely ViTaPEs . Our model captures both intra-modal structures (e.g., local texture in touch or scene context in vision) and cross-modal relationships, allowing for richer, fused representations. Transformers ( Vaswani et al. 2017 ) , while highly expressive, lack inherent inductive biases, relying instead on learning such biases through extensive training on large-scale datasets. However, in the visuotactile domain, such datasets are scarce, and training becomes particularly challenging due to the high-dimensional nature of multimodal inputs and the need for synchronized data collection. To address this, our model incorporates carefully designed positional encodings (PEs) that inject multi-scale spatial structure into the architecture, alleviating the need for prohibitively large datasets. Our model can be trained in both self-supervised and supervised regimes, facilitating task-agnostic embeddings while also optimizing for specific downstream tasks. Evaluations on out-of-distribution objects in real-world scenarios substantiate the efficacy of ViTaPEs in contact-rich tasks, object recognition, material characterization, texture analysis, and hardness assessment. We contribute with: • Multi-Scale Positional Encodings: Our ViTaPEs model employs a multi-scale positional encoding formulation to capture detailed spatial structures within each modality. Then it models cross-modal relationships to fuse tactile and visual data coherently, overcoming the inability of previous models to perform multiscale positional reasoning. • Theoretical Guarantees: We present ViTaPEs as the first visuotactile model that does not solely rely on intuition or empirical evidence alone, by providing theoretical analyses to support our experimental findings. Specifically, we show that our multi-scale visuotactile positional encodings are injective, rigid-motion–equivariant, and information-preserving, and we validate these properties empirically. • Zero-Shot Generalization and Transferable Robotic Representations: We demonstrate the out-of-distribution generalization capacity of ViTaPEs , trained with self-supervision, highlighting its robustness across diverse tasks and environments. We further show that ViTaPEs outperforms baseline methods on a real-world robotic grasping dataset, leveraging its transfer learning capabilities to adapt effectively to a smaller dataset of 10K samples. 2 Related Work Visuotactile Representation Learning. Visuotactile representation learning combines vision and touch to enhance perception in a wide range of tasks, including manipulation, material recognition, and texture analysis. Recent approaches have leveraged deep learning to jointly model visual and tactile data. Yuan et al. 2017b introduced a shared latent space for the two modalities using GelSight sensors ( Yuan et al. 2017a ) for fabric classification. Building on this, Yang et al. 2022 and Kerr et al. 2023b employed contrastive learning techniques to improve tactile representation learning with GelSight ( Yuan et al. 2017a ) and DIGIT ( Lambeta et al. 2020 ) sensors, respectively. Li et al. 2019 addressed the scale gap between visual and tactile signals using conditional adversarial networks to synthesize tactile data from visual inputs. Luo et al. 2018 improved cloth texture recognition by focusing on shared features across modalities, while the Visuo-Tactile Transformer (VTT) ( Chen et al. 2022 ) utilized spatial attention to effectively merge visual and tactile data. More recently, MViTac ( Dave et al. 2024 ) demonstrated the effectiveness of multimodal contrastive

Read the complete text