Query-Centric Diffusion Policy for Generalizable Robotic Assembly

Ziyi Xu, Haohong Lin, Shiqi Liu, Ding Zhao · 2025

Read Query-Centric Diffusion Policy for Generalizable Robotic Assembly by Ziyi Xu, Haohong Lin, Shiqi Liu, Ding Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-18686.txt.

Introduction

Assembling parts is one of the hardest jobs for a general robot, since contacts are fiddly and noise can ruin a fit. Ziyi Xu, Haohong Lin, Shiqi Liu, and Ding Zhao present the Query-centric Diffusion Policy in this 2025 paper. Robot assembly is usually split into high-level planning and low-level control. The two levels often mismatch in practice.

Their framework bridges the levels with queries that bundle the object, the contact point, and the skill to use. A query-centric step finds the task-relevant parts and guides the low-level policy. Point cloud views add robustness against noise. On the FurnitureBench benchmark, in simulation and on a real robot, the method improved skill precision and long-run success. For insertion and screwing, skill success rose by over fifty percent versus baselines. The queries, the authors say, are the bridge that was missing.

Catalog record

CREATORZiyi Xu, Haohong Lin, Shiqi Liu, Ding Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-18686

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Query-Centric Diffusion Policy for Generalizable Robotic Assembly Ziyi Xu, Haohong Lin, Shiqi Liu, Ding Zhao Abstract The robotic assembly task poses a key challenge in building generalist robots due to the intrinsic complexity of part interactions and the sensitivity to noise perturbations in contact-rich settings. The assembly agent is typically designed in a hierarchical manner: high-level multi-part reasoning and low-level precise control. However, implementing such a hierarchical policy is challenging in practice due to the mismatch between high-level skill queries and low-level execution. To address this, we propose the Query-centric Diffusion Policy (QDP), a hierarchical framework that bridges high-level planning and low-level control by utilizing queries comprising objects, contact points, and skill information. QDP introduces a query-centric mechanism that identifies task-relevant components and uses them to guide low-level policies, leveraging point cloud observations to improve the policy’s robustness. We conduct comprehensive experiments on the FurnitureBench in both simulation and real-world settings, demonstrating improved performance in skill precision and long-horizon success rate. In the challenging insertion and screwing tasks, QDP improves the skill-wise success rate by over 50% compared to baselines without structured queries. Fig. 1 : Overview of the furniture assembly task. The task nature comprises three parts: objects, contact points, and skill library. An effective assembly policy should identify the correct contact relationship given the assembly context and generate precise action sequences for the robot arm, bridging the gap between intra-part reasoning and inter-part precise control. I INTRODUCTION Contact-rich manipulation has been widely recognized as a critical task when building generalist intelligent robots [ 1 , 2 , 3 ] . In this field, robot assembly [ 4 , 5 ] stands out because it requires policies that are both precise and versatile to control the robot arm and interact with multiple objects. Despite the access to some offline demonstration data from human priors, robotic assembly poses two key challenges: inter-part relational reasoning and intra-part precise control in the online deployment. As is visualized in Figure 1 , the first challenge arises from the long-horizon, multi-part nature of the task, which demands accurate prediction of the next skill based on current observations, as well as identifying the correct objects for interaction. The second challenge becomes especially difficult in contact-rich scenarios where successful assembly hinges on precise object alignment. Even minor noise or occlusions in raw sensory observations can completely fail the sim-to-real transfer of low-level non-prehensile control policies. Various methods have been explored to address these challenges. One line of work focuses on high-level reasoning by leveraging cross-modality affordance-based approaches [ 6 ] , graph-based key point reasoning [ 7 ] , or skill-based retrieval [ 8 , 9 ] . However, these methods rely heavily on heuristic-based low-level controllers and predefined skill libraries, thus limiting their adaptability and precision. Meanwhile, recent advances in robot learning have significantly improved the precision and adaptability of low-level policies, thanks to (i) powerful imitation learning backbones such as diffusion models [ 10 , 11 , 12 ] and transformer-based architectures [ 13 , 14 ] , (ii) enhanced learning regimes like residual policies [ 15 ] and safe failure prevention methods [ 16 ] , and (iii) the integration of sensor modalities beyond vision and proprioception, such as tactile sensing [ 17 , 9 ] or point clouds [ 18 ] for improved contact modeling. Still, transferring these policies from simulation to the real world remains challenging due to the inherent difficulty of accurately simulating contact dynamics. In addition to the individual challenges of high-level and low-level policy design, integrating these two levels in a hierarchical framework introduces further complications. For instance, high-level policies may mis-specify objects or skills, causing the low-level policy to rely on the false queries that include irrelevant factors such as background pixels or non-impactful objects, ultimately leading to failure in real-world environments. Hence, it is crucial to establish a parsimonious proxy between high-level and low-level modules. To address these challenges and establish a robust interface between high-level and low-level modules, we propose Query-centric Diffusion Policy (QDP). Our framework leverages powerful pre-trained foundation models to extract high-level information by specifying both the requisite skills and target objects as a query. This query then serves as a powerful precondition to guide point cloud-based low-level control, forcing the robot agent to focus on the current task-relevant components when generating the action chunks under different contexts. Our contributions are threefold: • We introduce QDP, a query-centric diffusion policy framework that selects task-relevant queries for guiding low-level policies, enabling accurate skill selection and precise object interaction. • We propose a query-conditioned policy learning scheme to model the complex geometry captured by point cloud observations, improving precision and facilitating sim-to-real transfer. • We demonstrate the effectiveness of our approach on FurnitureBench in both simulation and real-world settings, showcasing robust performance under object misalignment and human intervention. II RELATED WORK Sequential decision making for robotic assembly Behavior learning methods are promising for addressing the combinatorial challenges of assembly sequence planning and fine-grained manipulation in robotic assembly tasks. Prior works have tackled the former problem by designing feasible assembly graphs [ 19 ] , disassembling strategies [ 20 ] , rearranging components through segmentation [ 21 ] , or even utilizing manual guidance [ 22 ] . While the latter challenge of manipulation involves precise control and adaptability, recent studies centered around FurnitureBench have employed contact-rich manipulation [ 9 ] , residual policies bridging sim-to-real gap [ 18 ] , [ 23 ] , fine-tuning diffusion policies [ 24 ] , or extracting skills from offline dataset [ 8 ] . Our approach tackles the two challenges by developing a unified hierarchical decision-making framework for robotic assembly. Hierarchical imitation learning Instead of learning the entire task using a single policy, we can decompose the long-horizon assembly problem into reusable sub-tasks, thereby introducing a hierarchical learning framework. This typically involves a high-level policy and a set of low-level policies. On the high-level decision-making side, approaches often rely on the observable conditions [ 25 ] and accurate system dynamics [ 26 ] to realize the precondition of applying skill and its effects. On the low-level side, previous skill-based reinforcement learning methods benefit from generalization through skill discovery but face challenges in skill chaining [ 27 ] and reward design [ 9 ] , especially in contact-rich environments. In this context, imitation learning can bridge the gaps between sub-tasks and avoid reward shaping by using full demonstration data. Previous data-driven hierarchical frameworks have included approaches such as sequence segmentation [ 28 ] , human-in-the-loop data collection systems [ 29 ] , and high-level planning based on heuristics [ 30 ] . Our focus, however, is on designing a hinge connecting high-level and low-level, generalizing across different assets and orders. Generative diffusion model for planning Recent advancements in diffusion-based generative strategies have led to widespread adoption in robotics, particularly in imitation learning [ 10 ] and planning [ 11 , 12 ] . These approaches have proven effective in capturing multi-modal distributions for planning and policy learning. Building on this diffusion-based framework, previous work explored learning visuomotor policy by integrating various conditions, including 3D representation [ 31 ] , equivariant models [ 32 ] , and affordance-based guidance [

Read the complete text