Kinematics-Aware Diffusion Policy with Consistent 3D Observation and Action Space for Whole-Arm Robotic Manipulation

Kangchen Lv*, Mingrui Yu*, Yongyi Jia, Chenyu Zhang, Xiang Li · 2025

Read Kinematics-Aware Diffusion Policy with Consistent 3D Observation and Action Space for Whole-Arm Robotic Manipulation by Kangchen Lv*, Mingrui Yu*, Yongyi Jia, Chenyu Zhang, Xiang Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-17568.txt.

Introduction

Kangchen Lv and coauthors teach robot arms to use the whole arm, not just the gripper, when they learn a task from demonstrations. Whole arm control matters when the arm must dodge obstacles with its elbow or push with its body. Yet most policies learn in joint space, which sits at a remove from the 3D world the task lives in.

Their framework describes the robot's state and its actions as a set of 3D points along the arm, the same form as the point cloud the robot sees. A diffusion policy then plans in that shared space, with the arm's kinematics built into the process so the planned motions stay feasible. An inverse kinematics solver then turns the plan into joint commands. In simulations and on a real arm the method reached higher success rates and adapted to new layouts better than prior approaches.

Catalog record

CREATORKangchen Lv*, Mingrui Yu*, Yongyi Jia, Chenyu Zhang, Xiang Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-17568

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Kinematics-Aware Diffusion Policy with Consistent 3D Observation and Action Space for Whole-Arm Robotic Manipulation Kangchen Lv*, Mingrui Yu*, Yongyi Jia, Chenyu Zhang, Xiang Li Abstract Whole-body control of robotic manipulators with awareness of full-arm kinematics is crucial for many manipulation scenarios involving body collision avoidance or body-object interactions, which makes it insufficient to consider only the end-effector poses in policy learning. The typical approach for whole-arm manipulation is to learn actions in the robot’s joint space. However, the unalignment between the joint space and actual task space (i.e., 3D space) increases the complexity of policy learning, as generalization in task space requires the policy to intrinsically understand the non-linear arm kinematics, which is difficult to learn from limited demonstrations. To address this issue, this letter proposes a kinematics-aware imitation learning framework with consistent task, observation, and action spaces, all represented in the same 3D space. Specifically, we represent both robot states and actions using a set of 3D points on the arm body, naturally aligned with the 3D point cloud observations. This spatially consistent representation improves the policy’s sample efficiency and spatial generalizability while enabling full-body control. Built upon the diffusion policy, we further incorporate kinematics priors into the diffusion processes to guarantee the kinematic feasibility of output actions. The joint angle commands are finally calculated through an optimization-based whole-body inverse kinematics solver for execution. Simulation and real-world experimental results demonstrate higher success rates and stronger spatial generalizability of our approach compared to existing methods in body-aware manipulation policy learning. Project Website: kinematics-aware-diffusion-policy.github.io I Introduction Imitation learning, where an agent learns to mimic the expert demonstrations, is an efficient approach to acquire complex manipulation skills from limited data. Recently, diffusion-based visual-motor policies [ 2 , 33 , 21 ] have shown many exciting results in imitation learning. Compared to traditional approaches, the remarkable abilities of diffusion models to learn multi-modal, high-dimensional action distributions are the key characteristics contributing to their success. Due to the alignment between the action space and task space which simplifies the policy learning process, Cartesian-space end-effector pose representations are widely used in existing imitation learning methods. However, for whole-arm robotic manipulation tasks, precise control over the full robot configuration is required, so imitating only the 6D end-effector pose trajectories is naturally insufficient. In many scenarios, such as operating in confined environments, avoiding collisions between the robot arm and surrounding obstacles is crucial. Additionally, certain tasks require the robot to interact with objects using parts of its body rather than the end-effector, further necessitating the whole-body control. Learning policies in joint space is a typical approach for whole-arm manipulation, which allows joint-level control of the entire configuration. However, joint space is inherently unaligned with the task space where the manipulation is conducted, forcing the policy to implicitly understand the complex non-linear kinematics. Thus, it is hard to learn a generalizable joint-to-task mapping from limited demonstrations, restricting the sample efficiency and spatial generalizability of joint-space policies. Fig. 1 : The proposed approach uses a set of 3D nodes on the arm body as both robot state and action representation for whole-arm manipulation, which is consistent with the 3D point cloud observation space and task space. Compared with using end-effector poses or joint angles, our method achieves higher spatial generalizability and sample efficiency while ensuring kinematic feasibility. To improve the policy learning performance for whole-body manipulation, some previous works explore to combine Cartesian space and joint space via incorporating differentiable kinematics within the policy networks [ 15 , 6 ] or concatenating redundant joint states upon the end-effector poses [ 17 ] . However, these methods still require the policy to predict joint-space actions, which cannot avoid the complexity brought by implicitly learning the non-linear kinematics. In this paper, we propose K inematics- A ware D iffusion P olicy ( KADP ), with consistent task, observation, and action spaces. Instead of using joint angles, both robot states and actions are represented with a set of 3D nodes on the robot arm body, making it convenient for the policy to infer the spatial and geometric relationship between the robot configuration and point cloud observations in the same 3D space. With such spatially consistent representation, the sample efficiency and spatial generalization of policy is improved while whole-body control is also enabled. To guarantee the kinematic feasibility of predicted 3D nodes, we further incorporate kinematic constraints into diffusion models. For execution, the joint angle commands are finally computed through an optimization-based whole-body inverse kinematics solver. In summary, the kinematics awareness of the proposed policy learning approach attributes to the following three aspects: 1. Whole-Arm Control : The proposed method enables manipulation over the entire robot configuration, overcoming the limitations of considering only Cartesian-space end-effector poses. 2. Consistent Task-Observation-Action Spaces : The node representation is in the 3D space, consistent with the observation and task spaces, allowing the policy to directly infer the spatial relationship between the arm body, objects, and environments. 3. Kinematic Feasibility Guarantee : By incorporating analytical joint-node mapping in both forward and reverse diffusion processes, our approach ensures that the generated node positions adhere to kinematic constraints. Across 8 simulation tasks on RLBench [ 9 ] and 4 real-world tasks, we systematically evaluate the performance of the proposed approach, with comparison to several baseline methods using different action representations. KADP achieves higher success rate and stronger spatial generalizability, suggesting the effectiveness of utilizing such 3D node-based robot state and action representation in body-aware manipulation learning. II Related Works II-A Diffusion Policies for Imitation Learning Diffusion models [ 24 , 8 ] are a class of probabilistic generative models that learn to generate samples from the prior distribution, typically a Gaussian distribution, by an iterative denoising process. For visual imitation learning from demonstrations, Diffusion Policy [ 2 ] pioneers the generation of actions through a conditional diffusion model. This innovative formulation is able to effectively learn the multi-modal distribution of demonstration actions while ensuring training stability, which has also been employed as action decoding head in many large-scale generalist policy models such as Octo [ 18 ] . Subsequently, many follow-up works are introduced to improve the generalization ability, data efficiency and inference speed of diffusion policies. DP3 [ 33 ] and 3D Diffusion Actor [ 12 ] enhance 3D scene representations by using 3D point cloud as observation space instead of RGB images, while some other works further leverage object-centric representations [ 13 ] or semantic fields [ 30 ] . In this paper, we also adopt 3D point cloud as it has been proved to be more effective than images. Beyond vanilla diffusion models, BESO [ 21 ] and PointFlowMatch [ 3 ] build policies upon score-based diffusion model and flow matching perspective, respectively. Besides, some works explore integrating several policies trained on heterogeneous data by composition [ 28 , 29 ] or accelerating diffusion policy with consistency distillation [ 14 , 20 ] . II-B Kinematics-Aware Policy Learning For robotic manipulation, the selection of action spaces, such as Cartesian space, joint space, and torque space, will greatly influence the performance of various downstream tasks [ 16 , 26 , 4 ] . Cartesian space, which controls the end-effector pose, is kinematics-unaware but aligns with the 3D Euclidean

Read the complete text