Modality Selection and Skill Segmentation via Cross-Modality Attention

Jiawei Jiang 1, Kei Ota 2, Devesh K. Jha 3, and Asako Kanezaki 1, Jiawei Jiang Science Tokyo, Kei Ota Mitsubishi Electric, Devesh K. Jha MERL, Asako Kanezaki Science Tokyo · 2025

Read Modality Selection and Skill Segmentation via Cross-Modality Attention by Jiawei Jiang 1, Kei Ota 2, Devesh K. Jha 3, and Asako Kanezaki 1, Jiawei Jiang Science Tokyo, Kei Ota Mitsubishi Electric, Devesh K. Jha MERL, Asako Kanezaki Science Tokyo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-14573.txt.

Introduction

Jiawei Jiang and co-authors tackle a 2025 robotics problem: extra senses can help a robot, but too much input at once can swamp learning. Their work covers touch and sound added to base robot models for hands on tasks. They propose cross-modality attention, a way to pick which sense is most useful at each step.

That picking method guides action choice by weighing the senses against each other over time. The same tool is then used to split expert demos into short primitive skills. Those skill parts train a layered policy, a control system with high and low levels. The authors aim it at long tasks that need rich contact, such as handling objects where touch tells more than sight alone at key moments.

Catalog record

CREATORJiawei Jiang 1, Kei Ota 2, Devesh K. Jha 3, and Asako Kanezaki 1, Jiawei Jiang Science Tokyo, Kei Ota Mitsubishi Electric, Devesh K. Jha MERL, Asako Kanezaki Science Tokyo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE15 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-14573

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Modality Selection and Skill Segmentation via Cross-Modality Attention Jiawei Jiang 1, Kei Ota 2, Devesh K. Jha 3, and Asako Kanezaki 1, Jiawei Jiang Science Tokyo, Kei Ota Mitsubishi Electric, Devesh K. Jha MERL, Asako Kanezaki Science Tokyo Abstract \finalcopy Abstract Incorporating additional sensory modalities such as tactile and audio into foundational robotic models poses significant challenges due to the curse of dimensionality. This work addresses this issue through modality selection. We propose a cross-modality attention (CMA) mechanism to identify and selectively utilize the modalities that are most informative for action generation at each timestep. Furthermore, we extend the application of CMA to segment primitive skills from expert demonstrations and leverage this segmentation to train a hierarchical policy capable of solving long-horizon, contact-rich manipulation tasks. 1 Introduction Recent advances in language and vision foundation models [ 1 , 2 , 3 ] have inspired the development of generalist foundation models for robotics [ 4 , 5 , 6 ] . These models are trained on large-scale datasets spanning diverse embodiments and tasks, demonstrating remarkable instruction-following capabilities. However, their manipulation performance—particularly in terms of precision and contact-rich interaction—remains limited. A key limitation is the lack of manipulation-specific sensory inputs, such as tactile data, which are crucial for fine-grained manipulation. Simply increasing the number of sensory modalities by stacking inputs often leads to exponentially larger policy search spaces, making learning inefficient and unstable. To address this, one line of work draws inspiration from Vision-Language Models (VLMs) [ 3 , 7 ] , projecting all modalities into the language domain to reduce search space dimensionality. While this improves generalization in simple scenarios like pick-and-place, it often sacrifices modality-specific granularity, making it unsuitable for fine-grained, contact-rich tasks. In contrast, we take inspiration from how humans manage multi-modal sensory inputs. Despite receiving a vast array of sensory signals, humans dynamically focus on the most relevant modalities depending on the task at hand. This selective attention is mirrored in the self-attention mechanisms of transformers, which can adaptively filter and prioritize information across modalities. In this paper, we explore this idea by applying a transformer-based multi-modal policy architecture to furniture assembly tasks [ 8 ] , which demand high accuracy and nuanced manipulation. Our goal is to improve the efficiency and accuracy of training multi-modal imitation learning policies for long-horizon, contact-rich tasks. We summarize our contributions as follows: • Cross-Modality Attention (CMA) : We propose a transformer-based architecture with a conditional U-Net backbone to learn CMA through imitation learning. • Unsupervised Primitive Segmentation : We demonstrate that CMA enables unsupervised segmentation of expert trajectories into primitive actions. • Improved Sample Efficiency : We show that training on segmented primitive actions leads to better accuracy and sample efficiency compared to end-to-end trajectory learning, due to more consistent input distributions within each primitive. 2 Related Work Diffusion Policy. Diffusion models [ 9 , 10 ] have emerged as a powerful class of generative models, revolutionizing data synthesis in domains such as image, audio, and more recently, robotics. Chi et al. [ 11 ] pioneered the application of conditional denoising diffusion to the robot action space, addressing key challenges in imitation learning—namely, multi-modal action distributions, temporal correlation, and the high precision demands of robotic tasks. This framework has since inspired a surge of research exploring variations of diffusion-based policies in robotics [ 12 , 13 ] . Further improvements have been proposed through goal-conditioned or reward-guided diffusion models [ 14 , 15 ] , aiming to inject task-specific guidance during action generation. Multi-modal training for Robotics. Recent work on generalist robotic models [ 4 , 5 , 6 ] has demonstrated the potential of using vision and language inputs to train policies capable of performing a wide range of manipulation tasks. These models, trained on large-scale datasets, exhibit impressive generalization. However, contact-rich manipulation remains a challenge due to its complexity and the limited information available from vision alone. To address this, a growing body of work has focused on multi-sensory learning. The ObjectFolder series [ 16 , 17 , 18 ] introduced simulated environments incorporating tactile and audio modalities. Li et al. [ 19 ] showed that incorporating such modalities improves performance in tasks like dense packing and water pouring. Similarly, Feng et al. [ 20 ] introduced a method that uses state tokens to dynamically adjust modality importance during task execution. Jones et al. [ 7 ] proposed a framework that enables multi-modal prompting by fine-tuning generalist models with tactile and audio inputs, extending their utility beyond vision and language. Skill Composition for long horizon tasks. Long-horizon manipulation tasks are often tackled by composing primitive actions. In traditional approaches, model-based frameworks such as behavior trees [ 21 ] and task-and-motion planning [ 22 , 23 , 24 ] explicitly define this structure. In contrast, learning-based methods [ 25 , 26 ] seek to learn hierarchical policies composed of a skill selector and a library of primitive skills. However, these approaches typically rely on human-labeled segmentation of demonstration trajectories, which is not only labor-intensive but also susceptible to ambiguity and inconsistency. Furniture Assembly. Furniture assembly has become a widely studied benchmark for long-horizon, contact-rich manipulation due to its complexity and structured sequence of operations. Heo et al. [ 8 ] introduced a standardized simulation environment and a suite of tasks to evaluate robotic performance in this domain. Building on this, Ankile et al. [ 13 ] identified bottleneck states in expert demonstrations to generate synthetic trajectories, enriching training data diversity. Other approaches include the use of hierarchical reinforcement learning, such as in Lin et al. [ 27 ] , which combines offline-trained skill transition models with skill-conditioned policies. Additionally, Ankile et al. [ 28 ] proposed a residual policy learning framework [ 29 ] to refine diffusion model outputs with reinforcement learning corrections. Despite these advancements, furniture assembly remains a challenging domain, underscoring the need for improved modeling of temporal structure, modality fusion, and generalization in contact-rich tasks. 3 Method To investigate our hypothesis that CMA can be used to segment primitive actions in an unsupervised fashion after sufficient training, we design a model that integrates CMA with a 1D-conditional U-Net diffusion policy. The goal is to leverage expert demonstrations via imitation learning and to train CMA to extract task-relevant features that can enhance policy performance in downstream contact-rich manipulation tasks. Our architecture consists of a 1D-conditional U-Net diffusion model as the policy backbone. We process state inputs using a fully connected layer and image inputs using a frozen R3M encoder [ 30 ] with a trainable projection layer. The resulting embeddings from each modality and timestep are stacked into a tensor of shape (T\cdot N\times B\times D , where T is the number of timesteps, N is the number of modalities, B is the batch size and D is the embedding dimension, which is passed through the CMA module. The output of the CMA serves as the conditional embedding for the diffusion model, which then generates action chunks for execution (see Fig. 1 ). Figure 1: Three stages design. First stage: train CMA using imitation learning. Second stage: Use CMA to cluster and segment primitive actions. Thrid stage: Train a hierarchical policy that select and execute individual primitive actions

Read the complete text