Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation

Yihang Zhu, Weiqing Wang, Shijie Wu, Ye Shi, Jingya Wang · 2025

Read Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation by Yihang Zhu, Weiqing Wang, Shijie Wu, Ye Shi, Jingya Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-18368.txt.

Introduction

Yihang Zhu leads a five author team teaching robots small named skills that stack into long jobs. Copy based robot training falters when many tasks share one brain: clumsy demos, clashing habits, and skills that blur together. AtomSkill first cuts demo runs into atomic skills, short moves with clear meaning, and lines them up across tasks by a contrast based rule.

Each skill also learns to picture its own end pose, the keypose where it should land. At run time a sampler strings skills together, and the pictured end poses trigger smooth hand offs. In both simulated and real trials, AtomSkill beat top copy based and skill based rivals on multi task work.

Catalog record

CREATORYihang Zhu, Weiqing Wang, Shijie Wu, Ye Shi, Jingya Wang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-18368

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation Yihang Zhu, Weiqing Wang, Shijie Wu, Ye Shi, Jingya Wang Abstract While imitation learning has shown impressive results in single-task robot manipulation, scaling it to multi-task settings remains a fundamental challenge due to issues such as suboptimal demonstrations, trajectory noise, and behavioral multi-modality. Existing skill-based methods attempt to address this by decomposing actions into reusable abstractions, but they often rely on fixed-length segmentation or environmental priors that limit semantic consistency and cross-task generalization. In this work, we propose AtomSkill, a novel multi-task imitation learning framework that learns and leverages a structured Atomic Skill Space for composable robot manipulation. Our approach is built on two key technical contributions. First, we construct a Semantically Grounded Atomic Skill Library by partitioning demonstrations into variable-length skills using gripper-state keyframe detection and vision-language model annotation. A contrastive learning objective ensures the resulting skill embeddings are both semantically consistent and temporally coherent. Second, we propose an Action Generation module with Keypose Imagination, which jointly predicts a skill’s long-horizon terminal keypose and its immediate action sequence. This enables the policy to reason about overarching motion goals and fine-grained control simultaneously, facilitating robust skill chaining. Extensive experiments in simulated and real-world environments show that AtomSkill consistently outperforms state-of-the-art methods across diverse manipulation tasks. $*$ $*$ footnotetext: Equal contributions. $\dagger$ $\dagger$ footnotetext: Corresponding authors. 1 Introduction Figure 1 : Comparison of skill learning strategies between previous skill-based imitation learning methods and our proposed AtomSkill. (a) Prior methods apply fixed-length sliding windows, resulting in overlapping motion fragments and ambiguous skill boundaries. (b) In contrast, AtomSkill segments demonstrations into semantically coherent and temporally aligned skills. Recent years have witnessed significant progress in imitation learning (IL) for robot manipulation 28 . Methods such as DP 7 , ACT 51 , and DP3 48 have demonstrated the ability to learn effective control policies directly from human teleoperated demonstrations. While these approaches excel in single-task settings, IL inherently struggles with fundamental challenges, including suboptimal expert demonstrations, trajectory noise, and behavioral multi-modality 34 ; 41 ; 52 . These issues become critically amplified in multi-task scenarios, where increased data diversity and complexity often lead to policies that fail to generalize or suffer from destructive interference across tasks 27 . Consequently, scaling imitation learning to a broad spectrum of tasks remains a fundamental and unresolved challenge. An intuitive approach for multi-task imitation learning is to structure manipulation behaviors by decomposing complex actions into reusable skill abstractions 39 ; 25 ; 35 . This line of work can be broadly categorized into two complementary strands. One focuses on learning compressed latent representations of skills that can be shared across tasks. For instance, methods such as 14 and 20 derive discrete skills from language paired with visual or privileged-state inputs. However, this reliance on environmental observations can bias the learned skills toward dataset- and scene-specific factors, limiting cross-task generalization. Another branch of action-only methods learns skills directly from raw action sequences: 44 employs a Vector Quantized Variational Autoencoder (VQ-VAE) 42 , while 33 and 27 enhance vector quantization with techniques such as finite scalar quantization 32 and rotation tricks 12 . Despite their representation efficiency, these methods typically rely on fixed-length sliding windows to segment demonstrations, which ignores the semantic structure of skills and conflicts with their variable-length nature 8 . As a result, the learned latent tokens often capture short, repetitive motion fragments rather than semantically meaningful and reusable skills, ultimately limiting their compositional generality and applicability. To address these challenges, we propose AtomSkill , a multi-task imitation learning framework that learns and leverages an Atomic Skill Library for cross-task robot manipulation. Unlike approaches that rely on fixed-length segmentation, AtomSkill partitions demonstrations into variable-length, semantically meaningful skills by detecting keyframes from gripper state changes and leveraging a large Vision-Language Model (VLM) 50 to annotate each segment. Our approach is built around two core innovations. First, we introduce Atomic Skill Learning with Semantic Contrastive Skill Alignment, which segments demonstrations into variable-length atomic skills and employs a contrastive objective to jointly ensure both semantic consistency and temporal coherence in the learned skill embeddings. Our framework learns a VQ-VAE-style skill prior regularized by a contrastive objective, forming a compact and semantically grounded codebook. Second, our Action Decoder with Keypose Imagining jointly predicts both the skill’s terminal keypose and the immediate action sequence, enabling the policy to reason about long-horizon motion intent and fine-grained control simultaneously. Here, we introduce a novel inference paradigm driven by a Skill Diffusion Sampler, which generates plausible skill sequences from the library for robust high-level planning. This sampler is coupled with Action Chunking with Keypose, where the predicted keypose acts as a progress monitor to autonomously trigger smooth skill transitions, enabling reliable long-horizon execution without manual heuristics. Extensive experiments in simulation and real-world environments demonstrate that AtomSkill consistently outperforms state-of-the-art imitation learning and skill-based baselines. Ablation studies confirm the contribution of each component, highlighting the importance of semantic-temporal alignment and keypose-conditioned chunking. In summary, our contributions are as follows: • AtomSkill Framework for Multi-Task Robot Manipulation. We introduce AtomSkill, a novel multi-task imitation learning framework that learns and leverages a structured Atomic Skill Space to enable composable robot manipulation. • Semantically Grounded Atomic Skill Library. We develop a skill learning paradigm that partitions demonstrations into variable-length atomic skills using gripper-state keyframe detection and vision-language model annotation. A contrastive learning objective ensures both semantic consistency and temporal coherence, creating a compact and reusable skill codebook. • Action Generation with Keypose Imagination. We design an action decoder that jointly predicts terminal keyposes (long-horizon intent) and immediate actions. This keypose imagination mechanism enables simultaneous reasoning about motion goals and fine-grained control, facilitating robust skill chaining and spatial understanding. 2 Related Work 2.1 Multi-Task Learning in Robotic Manipulation As a long-standing central challenge in robotics, multi-task learning has been extensively studied 9 ; 36 ; 15 . A particularly prominent recent direction is the Vision-Language-Action model 34 ; 40 ; 23 ; 4 ; 17 ; 31 , which typically involves training large-scale networks on massive datasets. While these models demonstrate impressive multi-task capabilities, collecting such vast amounts of data in robotics is extremely time-consuming and resource-intensive 10 . Consequently, significant research efforts have been dedicated to enhancing model performance under data-limited conditions. One line of work addresses this through carefully designed network architectures. For instance, PerAct 38 , 3D Diffuser Actor 21 , and GNFactor 47 utilize RGB-D inputs by projecting 2D features into 3D voxel space to learn fused 2D-3D representations, which then serve as input to transformer-based action predictors. Alternatively, Sparse Diffusion Policy 43 and Factorized Diffusion Policy 30 draw inspiration from Mixture-of-Experts 37 ; 18 , employing modular networks where each expert or module implicitly learns different skills from the dataset. Beyond architectural innovations, another category of research leverages Latent Variable Models to learn a structured latent space from data, aiming to uncover shared knowledge across different tasks for more efficient data utilization. 2.2 Latent Variable Model for Decision Making Latent Variable Models (LVMs) have proven highly effective for acquiring structured representations in both offline reinforcement learning (RL) 2 ; 24 ; 35 ; 53 ; 5 and imitation learning (IL) 29 ; 14 ; 26 . 24

Read the complete text