Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation

Shengqi Xu 1,2,3, Guojin Zhong 1,2, Yang Liu 1,2, Fanjie Wang 1,2, Hu Luo 1,2, Hanyu Zhou 4, Weiyao Zhang 1,2 Ziyi Ye 1,2, Zuxuan Wu 1,2,3, † \dagger, Yu-Gang Jiang 1,2, † \dagger · 2026

Read Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation by Shengqi Xu 1,2,3, Guojin Zhong 1,2, Yang Liu 1,2, Fanjie Wang 1,2, Hu Luo 1,2, Hanyu Zhou 4, Weiyao Zhang 1,2 Ziyi Ye 1,2, Zuxuan Wu 1,2,3, † \dagger, Yu-Gang Jiang 1,2, † \dagger (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29941.txt.

Introduction

Xu and nine colleagues build a robot policy that unites vision and touch for contact-rich manipulation. The tasks include erasing a whiteboard or sharpening a pencil, where force matters and cameras alone cannot feel contact.

The team's key finding is that the correlation between momentary and accumulated tactile motion sharply distinguishes fine contact states. Their ViTacMotor policy uses this tactile motion correlation as a motion-aware touch representation. A mixture-of-transformers design keeps a vision-language expert for understanding and a generative expert for action, while letting the senses inform each other. In experiments the policy perceives contact states and force levels that raw tactile images miss, and it outperforms earlier visuo-tactile policies across the tasks tested.

Catalog record

CREATORShengqi Xu 1,2,3, Guojin Zhong 1,2, Yang Liu 1,2, Fanjie Wang 1,2, Hu Luo 1,2, Hanyu Zhou 4, Weiyao Zhang 1,2 Ziyi Ye 1,2, Zuxuan Wu 1,2,3, † \dagger, Yu-Gang Jiang 1,2, † \dagger
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29941

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Seeing Touch from Motion: A Unified Modality-Aware Visuo-Tactile Policy with Tactile Motion Correlation Shengqi Xu 1,2,3, Guojin Zhong 1,2, Yang Liu 1,2, Fanjie Wang 1,2, Hu Luo 1,2, Hanyu Zhou 4, Weiyao Zhang 1,2 Ziyi Ye 1,2, Zuxuan Wu 1,2,3, † \dagger, Yu-Gang Jiang 1,2, † \dagger Abstract 1] Institute of Trustworthy Embodied AI, Fudan University 2] Shanghai Key Laboratory of Multimodal Embodied AI 3] NeoteAI 4] School of Computing, National University of Singapore \checkdata [Website] https://shengqi77.github.io/Seeing-Touch-from-Motion/ † † footnotetext: † Corresponding authors. 1 Introduction Figure 1 : Comparison of three different tactile representations for distinguishing fine-grained contact states. (a) Fine-grained contact states, such as making contact ( contact state B ) and releasing contact ( contact state A ), can be visually similar. (b) Tactile raw appearance primarily captures visual changes but suffers from ambiguity due to their deceptive similarity, making them difficult to distinguish. (c) Tactile cumulative motion (optical flow between the no-contact frame and the current frame) reflects overall deformation but still exhibits similar patterns, making it equally difficult to distinguish. (d) In contrast, our tactile motion correlation explicitly resolves this ambiguity by analyzing the dynamic relationship between transient and cumulative motion. The directional correlation (same v.s. opposite) effectively breaks the similarity, providing highly discriminative cues crucial for contact-rich manipulation. Contact-rich manipulation is crucial for robots performing real-world fine-grained tasks, such as precision insertion and object wiping, where success hinges on accurately perceiving subtle physical interactions. Such tasks typically rely on the complementary cooperation of global visual perception and local tactile feedback. Specifically, vision provides global contextual information to support the planning of coarse motion trajectories, while touch captures local contact feedback to enable fine-grained interaction with objects. To capture local contact feedback, a series of tactile sensing technologies has been developed. Among these, optical tactile sensors [ 60 , 12 , 51 , 66 , 53 ] are widely adopted due to their high spatial resolution, texture perception and multi-dimensional force sensing capabilities. These advantages arise from their imaging mechanism, where an internal camera is utilized to monitor the deformation of an elastic gel surface. The surface is typically patterned with markers to better present gel motion during external contacts. Although optical tactile sensors offer information-rich visual measurements that indirectly reflect tactile cues, effectively extracting fine-grained contact states from these cues remains an underexplored problem. Most existing methods [ 57 , 62 , 54 ] typically rely on either raw images or cumulative motion fields ( i.e. , optical flow between the current frame and the initial non-contact frame) to represent tactile information. However, both representations struggle to distinguish fine-grained contact states, such as the distinction between making contact and releasing contact (see Fig. 1 (a)), which is crucial for contact-rich manipulation. As shown in Fig. 1 (b), raw images primarily capture appearance changes caused by extern contact but suffer from perceptual ambiguity due to their deceptive similarity, making them difficult to distinguish. Similarly, Figure 1 (c) illustrates that cumulative tactile motion only reflects the direction and magnitude of overall gel deformation but still exhibits similar patterns, making it equally difficult to distinguish subtle contact variations. To address the above issue, we thoroughly explore the underlying motion priors in tactile sensing and discover that the correlation between transient motion and cumulative motion can more explicitly distinguish fine-grained contact states . Specifically, transient motion ( i.e. , optical flow between consecutive frames) captures instantaneous deformation, while cumulative motion captures the overall deformation relative to the initial non-contact state. We observe that different contact states exhibit distinct correlation patterns between these two types of motion. For instance, Figure 1 (d) illustrates that when releasing contact, the motion vectors point in opposite directions, whereas when making contact, they align in the same direction (See Fig. 3 (a) for more examples). These observations inspire us to propose Tactile Motion Correlation (TMC) , a motion-aware tactile representation that explicitly captures fine-grained contact states by modeling the correlation between transient and cumulative motions through their dot product . This tactile representation offers three main advantages. First, the sign and magnitude of the dot product enable precise differentiation among fine-grained contact states, while carrying a physically interpretable meaning of the underlying contact. Second, the magnitude exhibits a strong positive correlation with the applied force, thereby faithfully reflecting contact intensity. Third, it is a sensor-agnostic representation compatible with various optical sensors, allowing it to bridge heterogeneous sensors and holds promise for enabling a general tactile representation in the future. While an informative tactile representation is essential for contact-rich manipulation, the effective integration of local touch with global vision is equally crucial. Existing fusion methods mainly fall into two categories: feature-level fusion methods [ 36 , 71 , 58 ] and decision-level fusion methods [ 6 ] . The former typically concatenates visual and tactile features or applies simple attention mechanisms directly to them. However, such strategies overlook the distinct properties of each modality and may allow one modality to dominate the other. In contrast, the latter typically train an independent policy network for each modality and fuses their outputs. Though it respects modality-specific differences, the absence of interaction between vision and touch prevents the policy from exploiting cross-modal cues, making it difficult to achieve genuine complementarity. In this work, we take advantage of the Mixture-of-Transformers (MoT) architecture [ 41 ] and propose ViTacMotor , a unified yet modality-aware Vi suo- Tac tile policy that incorporates the proposed Mot ion-aware c or relation representation. The MoT adopted in ViTacMotor decouples modality-specific parameters to preserve the unique properties of each modality, while enabling effective cross-modal interaction via a shared global self-attention mechanism. This design enables ViTacMotor to capture cross-modal complementarity while respecting the individuality of visual and tactile inputs, thereby facilitating effective integration of local touch with global vision for contact-rich manipulation. Our main contributions are summarized as follows: 1. We reveal that the correlation between transient and cumulative tactile motion can more explicitly distinguish fine-grained contact states. Motivated by this finding, we propose Tactile Motion Correlation , a motion-aware representation that addresses the perceptual ambiguity in prior arts and thus enable fine-grained contact-state perception for facilitating contact-rich manipulation. 2. We propose ViTacMotor , a unified yet modality-aware visuo-tactile policy framework. Compared to existing methods, it can capture cross-modal interactions while preserving the unique properties of each modality, enabling an effective integration of global visual perception and local tactile sensing. 3. We compare our method with existing methods on a variety of challenging contact-rich manipulation tasks. Extensive experiments demonstrate the superior performance and robustness of our method. 2 Related Work Tactile Sensing in Robotics is crucial for acquiring rich microscopic contact feedback during interactions with objects that complements macroscopic visual perception [ 50 , 30 , 39 ] . Tactile sensors aim to transduce external physical contact into measurable electrical signals. They can be categorized into various types based on the transduction mechanisms, including piezoelectric [ 68 , 24 , 32 ] , capacitive [ 45 , 46 ] , magnetic [ 29 , 2 ] , and optical sensors [ 66 , 43 , 53 , 42 , 52 , 63 , 56 , 23 ] . In particular, optical tactile sensors have been widely

Read the complete text