A Novel Task-Driven Diffusion-Based Policy with Affordance Learning for Generalizable Manipulation of Articulated Objects
Hao Zhang, Zhen Kan, Weiwei Shang, Yongduan Song · 2025
Read A Novel Task-Driven Diffusion-Based Policy with Affordance Learning for Generalizable Manipulation of Articulated Objects by Hao Zhang, Zhen Kan, Weiwei Shang, Yongduan Song (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-14939.txt.
Introduction
Hao Zhang, Zhen Kan, Weiwei Shang, and Yongduan Song introduce DART for manipulating hinged and jointed objects in a 2025 robotics paper. Opening doors, drawers, and lids requires finding the right contact points and motion, and generalizing across object categories is still hard. DART strengthens a diffusion-based policy with two extra ingredients.
Linear temporal logic descriptions give the policy a clear representation of task semantics, the ordered steps a job requires. Affordance learning identifies the best places to interact with each object. An optimization step based on interaction data then refines the proposed actions, going beyond what demonstration-trained diffusion policies usually achieve. Experiments show DART ahead of most existing methods on manipulation success, generalization to new categories, and transfer of reasoning between tasks.
Catalog record
| CREATOR | Hao Zhang, Zhen Kan, Weiwei Shang, Yongduan Song |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 66 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-14939
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Novel Task-Driven Diffusion-Based Policy with Affordance Learning for Generalizable Manipulation of Articulated Objects Thanks: Hao Zhang, Zhen Kan (Corresponding Author), and Weiwei Shang are with the Department of Automation at the University of Science and Technology of China, Hefei, Anhui, China. Yongduan Song is with the School of Automation, Chongqing University, Chongqing, China. Thanks: This work was supported in part by National Key R&D Program of China under Grant 2022YFB4701400/4701403 and National Natural Science Foundation of China under Grant U201360. Hao Zhang, Zhen Kan, Weiwei Shang, Yongduan Song Abstract Despite recent advances in dexterous manipulations, the manipulation of articulated objects and generalization across different categories remain significant challenges. To address these issues, we introduce DART, a novel framework that enhances a d iffusion-based policy with a ffo r dance learning and linear t emporal logic (LTL) representations to improve the learning efficiency and generalizability of articulated dexterous manipulation. Specifically, DART leverages LTL to understand task semantics and affordance learning to identify optimal interaction points. The diffusion-based policy then generalizes these interactions across various categories. Additionally, we exploit an optimization method based on interaction data to refine actions, overcoming the limitations of traditional diffusion policies that typically rely on offline reinforcement learning or learning from demonstrations. Experimental results demonstrate that DART outperforms most existing methods in manipulation ability, generalization performance, transfer reasoning, and robustness. For more information, visit our project website at: https://sites.google.com/view/dart0257/. I Introduction The manipulation of articulated objects has been an interesting and important topic in robotic learning. Although prior research has demonstrated promising results in the manipulation of rigid bodies, significant challenges persist when it comes to handling articulated objects [ 1 ] . Generalizing to various types of articulated objects [ 2 ] is particularly difficult for dexterous manipulations. For example, if a dexterous hand can open the lid of a toilet, it should also be capable of opening the lid of a garbage can, despite their cosmetic differences. While many recent efforts have focused on improving the robotic generalization performance [ 3 ] or reducing the exploration burden [ 4 ] , enhancing the learning efficiency or improve the generalization ability for high degrees of freedom (DOF) skills, such as dexterous manipulation, remains a challenging problem, not to mention achieving both simultaneously. There are various methods enabling skill generalization via reinforcement learning (RL). Taking advantage of Meta-RL to improve the generalization potential of algorithms is an effective approach [ 5 ] . However, solving context Markov decision processes (MDPs) in Meta-RL often involves cumbersome computation. Recently, increasing efforts are being dedicated to leveraging vision features to enhance RL in tackling various challenging tasks [ 6 , 7 , 8 ] . One of the interesting findings is that algorithms exploiting point cloud information can have good generalization without any Meta-RL-like training [ 9 ] . However, most of the previous work struggles to understand the geometry of articulated objects, thus making it challenging to identify appropriate interaction points when developing manipulation skills. One of the effective ways to address the above shortcomings is affordance learning [ 10 ] . In [ 11 ] , a learning-from-interaction framework was proposed to manipulate articulated objects, which uses an actionability scoring module to estimate the score of each pixel and provide better interaction positions. The work of [ 12 ] further extends [ 11 ] by exploiting the multi-stage stable learning and self-supervised data collection to achieve deformable object manipulation. In [ 13 ] , RL was guided by affordance learning to achieve diverse manipulation, which yields an end-to-end learning framework to avoid offline data collection. Although affordance learning has shown great potential in the manipulation of articulated objects, generating a high-quality affordance map often requires a significant amount of interactions for training, which significantly impacts learning efficiency in long-horizon tasks. Due to the rich expressivity, linear temporal logic (LTL) [ 14 ] is capable of describing long-horizon tasks composed of logically organized sub-tasks. For instance, LTL specification was converted to a finite-state predicate automaton in [ 15 ] to facilitate the use of RL in performing complex manipulation tasks. However, previous methods mainly focus on solving tasks with different automaton, without considering the task semantics of LTL instructions. Based on graph neural networks, an approach exploiting the compositional syntax and semantics of LTL was proposed in [ 16 ] to learn task-conditioned policies, which is generalizable to new instructions. In [ 17 ] , Transformer [ 18 ] was employed to encode the LTL specifications for improved representation capability and interpretability. Direct language approaches that utilize large language models [ 19 ] or represent tasks using text [ 20 ] have brought success to manipulation at the task level. However, these approaches are limited either by their expressiveness and rigor, or by the computational efficiency of the algorithms relative to LTL. Despite recent progress, neither vision-based RL nor LTL-guided motion planning inherently improves the generalization of the learned manipulation skills. As an effective generative model, the diffusion model (DM) [ 21 , 22 , 23 ] utilizes a parametric reverse process to generate samples from a Gaussian distribution. Inspired by this, DM has been investigated for long-horizon manipulation [ 24 , 25 , 26 ] . However, in most of these studies, DM has functioned as a policy within either an offline reinforcement learning or imitation learning framework. Extending this approach to dexterous manipulations is challenging due to the difficulty in obtaining high-quality operational data. Our primary motivation is to determine whether task semantics can be leveraged to drive affordance learning and guide the agent in achieving a policy with the diffusion model, thereby improving learning efficiency for articulated objects and enhancing generalization in operational performance. To address this, we propose DART, a novel policy generalization framework that enhances a d iffusion-based policy using a ffo r dance learning and linear t emporal logic representations to improve both generalizability and learning efficiency. DART guides the agent in improving sampling efficiency by encoding task semantics into LTL representations. Additionally, it utilizes affordance learning to generate an affordance map based on point cloud information captured by the camera, predicting optimal contact points for dexterous manipulation. Furthermore, DART incorporates a task-driven action optimization method for the policy, enabling training and optimization in online settings. Contributions: The main contributions of this work are summarized as follows: • We introduce a novel policy generalization algorithm that enhances a diffusion-based policy through the use of LTL representations and affordance learning, thereby improving learning efficiency and ensuring robust generalization performance. • We construct a task-driven MDP based on LTL representations, which not only incorporates task semantics to facilitate robotic learning but also designs a contact planner based on affordance learning to enhance the efficiency of articulated manipulations. • We exploit a diffusion-based policy for interacting with articulated objects, whose multimodal policy representation lays the groundwork for the generalization. • We further extend the task environment of [ 9 ] and evaluate the methods when generalizing to different category-level objects. A comprehensive empirical study validates that DART significantly outperforms baseline methods in terms of manipulation ability, generalization performance, transfer reasoning, and robustness. II Preliminaries and Problem Formulation II-A sc-LTL and LTL Progression Co-safe LTL (sc-LTL)