Structural Action Transformer for 3D Dexterous Manipulation

Xiaohan Lei, Min Wang, Bohong Weng, Wengang Zhou, Houqiang Li, [6pt] MoE Key Laboratory of Brain-inspired Intelligent Perception, CognitionUniversity of Science and Technology of China · 2026

Read Structural Action Transformer for 3D Dexterous Manipulation by Xiaohan Lei, Min Wang, Bohong Weng, Wengang Zhou, Houqiang Li, [6pt] MoE Key Laboratory of Brain-inspired Intelligent Perception, CognitionUniversity of Science and Technology of China (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03960.txt.

Introduction

Xiaohan Lei and others propose the Structural Action Transformer, a 3D dexterous manipulation policy tackling cross-embodiment skill transfer for high-degree-of-freedom robotic hands. The policy reframes each action chunk as an unordered sequence of joint-wise trajectories, letting a Transformer handle varied joint counts. An Embodied Joint Codebook encodes each joint functional role and kinematic traits to supply structural priors and resolve ambiguity. Pretraining on large heterogeneous data plus test and real-world fine-tuning beats all rivals.

Catalog record

CREATORXiaohan Lei, Min Wang, Bohong Weng, Wengang Zhou, Houqiang Li, [6pt] MoE Key Laboratory of Brain-inspired Intelligent Perception, CognitionUniversity of Science and Technology of China
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39794
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.03960

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Structural Action Transformer for 3D Dexterous Manipulation Xiaohan Lei, Min Wang, Bohong Weng, Wengang Zhou, Houqiang Li, [6pt] MoE Key Laboratory of Brain-inspired Intelligent Perception, CognitionUniversity of Science and Technology of China Abstract Achieving human-level dexterity in robots via imitation learning from heterogeneous datasets is hindered by the challenge of cross-embodiment skill transfer, particularly for high-DoF robotic hands. Existing methods, often relying on 2D observations and temporal-centric action representation, struggle to capture 3D spatial relations and fail to handle embodiment heterogeneity. This paper proposes the Structural Action Transformer (SAT), a new 3D dexterous manipulation policy that challenges this paradigm by introducing a structural-centric perspective. We reframe each action chunk not as a temporal sequence, but as a variable-length, unordered sequence of joint-wise trajectories. This structural formulation allows a Transformer to natively handle heterogeneous embodiments, treating the joint count as a variable sequence length. To encode structural priors and resolve ambiguity, we introduce an Embodied Joint Codebook that embeds each joint’s functional role and kinematic properties. Our model learns to generate these trajectories from 3D point clouds via a continuous-time flow matching objective. We validate our approach by pre-training on large-scale heterogeneous datasets and fine-tuning on simulation and real-world dexterous manipulation tasks. Our method consistently outperforms all baselines, demonstrating superior sample efficiency and effective cross-embodiment skill transfer. This structural-centric representation offers a new path toward scaling policies for high-DoF, heterogeneous manipulators. Figure 1 : Conceptual illustration of action chunk tokenization. (a) The conventional temporal-centric perspective, which structures actions as a sequence of T timesteps (chunk length), with each token having dimension D_{a} (action dim). (b) Our proposed structural-centric perspective, which reframes the action chunk as a sequence of D_{a} joints, where each token’s feature is its temporal trajectory over T . This (D_{a},T) view naturally handles heterogeneous embodiments as a variable-length, unordered sequence, which is a key feature of our approach. 1 Introduction The quest for robotic systems capable of human-level dexterity represents a grand challenge in Embodied Artificial Intelligence. Dexterous robotic hands, with their high degrees of freedom (DoF), hold the potential to perform a vast array of complex, contact-rich manipulation tasks currently beyond the reach of simpler parallel-jaw grippers. Imitation learning, particularly in its offline formulation, has emerged as a promising paradigm for teaching such skills by leveraging large-scale datasets of human demonstrations 45 ; 63 ; 28 ; 13 ; 66 ; 41 ; 15 ; 27 ; 8 . A critical bottleneck, however, lies in the cross-embodiment transfer of these skills: how can a robot effectively learn from heterogeneous demonstrations, given the significant differences in morphology, kinematics, and sensory feedback 68 ; 85 ; 9 ; 75 ? While recent Vision-Language-Action (VLA) models have made strides in general-purpose robotics 87 ; 83 ; 29 ; 72 ; 79 , they predominantly rely on 2D visual inputs, which often fail to capture the intricate 3D spatial relationships essential for precise dexterous manipulation 51 ; 32 ; 31 ; 22 ; 81 . This work tackles the challenge of cross-embodiment imitation for dexterous hands by directly learning from 3D point cloud observations, proposing a fundamental shift in how we represent and process robotic actions. A dominant paradigm in recent policy learning is action chunking, where a model predicts a sequence of future actions (T,D_{a}) 12 ; 80 . As in Figure 1 (a), this temporal-centric perspective , which treats each D_{a} -dimensional vector as a token in a temporal sequence, is effective for low-dimensional systems. However, this representation faces a fundamental challenge when scaling to high-DoF, heterogeneous manipulators. As the action dimension D_{a} grows ( e.g. , from a 7-DoF robot arm to a 24-DoF dexterous hand), the model must learn complex, implicit correlations within a monolithic feature vector. More critically, this fixed-dimensional view provides no natural mechanism for cross-embodiment transfer, as different morphologies cannot be directly compared. This paper challenges this conventional wisdom by reframing the action representation problem. We propose a structural-centric perspective , modeling actions as a sequence of D_{a} embodied joints, where each joint’s feature is its trajectory over the time horizon T , i.e. , (D_{a},T) , as shown in Figure 1 (b). This structural view directly addresses the heterogeneity problem: the sequence length D_{a} can now vary between embodiments, a property that Transformer architectures handle natively 65 . This allows a policy to learn transferable skills by finding functional similarities between corresponding joints. At the same time, this formulation also allows the model to learn a compressed representation of motion primitives for each joint, as time is now treated as the feature dimension. We introduce the Structural Action Transformer (SAT), a novel 3D dexterous manipulation policy built upon this structural-centric representation. Our model takes raw 3D point cloud observations and natural language instructions as input. The 3D scene is processed by a hierarchical tokenizer that uses Farthest Point Sampling (FPS) and PointNets 50 to extract both local geometric tokens and a single global scene token. These are combined with language features from a T5 encoder 54 to form a multi-modal observation sequence. This sequence conditions a Diffusion Transformer (DiT) 47 that operates directly on our (D_{a},T) structural action representation. We treat the D_{a} joints as a variable-length sequence, where each token represents a compressed temporal trajectory for a single joint. To explicitly encode structural priors and manage heterogeneity, we introduce an Embodied Joint Codebook. This codebook provides learnable embeddings for each joint based on its morphological properties, enabling the model to identify functional correspondences across different embodiments. The policy is trained to generate the entire action chunk by learning a conditional velocity field via a continuous-time flow matching objective 34 , with the final action produced by an ODE solver. We demonstrate the efficacy of this approach by pre-training on large-scale, heterogeneous datasets of both human and robot demonstrations 37 ; 18 ; 46 ; 16 ; 67 and then fine-tuning it on a suite of challenging simulation benchmarks 55 ; 2 ; 10 and real-world bimanual manipulation tasks. Our extensive experiments show that this structural-centric paradigm consistently outperforms strong baselines 12 ; 68 ; 82 ; 77 ; 74 and achieves superior sample efficiency, proving its effectiveness in cross-embodiment skill transfer. Fundamentally, our work is the first to successfully implement a policy that tokenizes actions along the structural dimension, offering a new and scalable path toward learning generalist policies for a diverse ecosystem of high-DoF, heterogeneous manipulators. 2 Related Work 2.1 Action Representation in Policy Learning The representation of actions is a fundamental design choice in learning-based control. While early methods in imitation learning predict a single action at each timestep 49 ; 59 , this autoregressive approach is known to suffer from compounding errors, where small inaccuracies accumulate over long horizons, leading to significant trajectory divergence 58 . To mitigate this issue, a dominant paradigm in recent literature is action chunking , where a policy outputs a sequence of future actions at each inference step 61 ; 80 ; 12 . This approach has been shown to improve temporal consistency and reduce the impact of compounding errors. For instance, Diffusion Policy 12 and related diffusion-based models 25 ; 57 ; 70 ; 21 ; 36 ; 35 generate chunked action trajectories

Read the complete text