BrickCraft : Visuomotor Skill Composition with Situated Manual Guidance for Long-Horizon Interlocking Brick Assembly

Jichuan Yu, Bowei Li, Zhenran Tang, Guanxing Lu, Chuxiong Hu, Ruixuan Liu, Changliu Liu · 2026

Read BrickCraft : Visuomotor Skill Composition with Situated Manual Guidance for Long-Horizon Interlocking Brick Assembly by Jichuan Yu, Bowei Li, Zhenran Tang, Guanxing Lu, Chuxiong Hu, Ruixuan Liu, Changliu Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07605.txt.

Introduction

Jichuan Yu and six co-authors present BrickCraft, a robot framework for assembling interlocking bricks over long task horizons. Building with bricks mixes big-picture planning, precise spatial grounding, and delicate manipulation, and each new structure seems to demand new skills. BrickCraft instead breaks assembly into reusable primitive moves.

Each step is described relative to a reference brick already in the partial structure, which turns open-ended builds into combinations of a finite skill set. Situated manuals project the intended placement onto the robot's live camera view, guiding the learned skills during execution. Experiments show the system learned proficient assembly from few demonstrations and recombined its skills across diverse long-horizon tasks.

Catalog record

CREATORJichuan Yu, Bowei Li, Zhenran Tang, Guanxing Lu, Chuxiong Hu, Ruixuan Liu, Changliu Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.07605

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BrickCraft : Visuomotor Skill Composition with Situated Manual Guidance for Long-Horizon Interlocking Brick Assembly Thanks: Indicates equal contribution. Jichuan Yu, Bowei Li, Zhenran Tang, Guanxing Lu, Chuxiong Hu, Ruixuan Liu, Changliu Liu Abstract Autonomous robotic assembly of interlocking bricks demands seamless integration of long-horizon task reasoning, spatial grounding, and fine-grained manipulation. This paper presents BrickCraft , a compositional framework designed for long-horizon and generalizable interlocking brick assembly. BrickCraft models the assembly process using a relative formulation, where each step is anchored to a reference brick within the partial structure, thereby decomposing complex tasks into a finite set of reusable primitive skills. BrickCraft bridges the gap between high-level assembly plans and physical execution through situated manuals, which provide explicit spatial guidance for learned visuomotor skills by projecting the assembly intent onto real-time robot observations. Finally, BrickCraft employs a compositional execution pipeline that chains these spatially grounded skills to accomplish long-horizon assembly tasks. Extensive experimental validations demonstrate that BrickCraft acquires proficient assembly skills from a limited set of demonstrations and exhibits strong compositional generalization to unseen structures. The project website is available at https://intelligent-control-lab.github.io/BrickCraft . I INTRODUCTION Autonomous long-horizon assembly remains a formidable open problem in robotics, as it necessitates the seamless integration of physically grounded task reasoning and dexterous, contact-rich manipulation [ 13 , 22 ] . Taking the assembly of interlocking bricks as an example, a robot must not only reason about structural dependencies to synthesize a feasible assembly plan but also perform fine-grained actions under tight geometric constraints. Mastering such capabilities represents a fundamental step toward intelligent embodied systems in complex real-world settings. While recent advances in learning-based approaches have demonstrated remarkable proficiency in dexterous manipulation and impressive generalization [ 3 , 2 ] , monolithic end-to-end control paradigms for long-horizon assembly remains largely intractable due to the inherent difficulty of implicitly modeling long-term dependencies and extensive data requirements. In contrast, hierarchical frameworks offer a viable alternative by decoupling task reasoning from execution [ 24 , 27 ] . Operating much like humans following an assembly manual, these approaches first employ a task planner to decompose the task into a sequence of actionable subgoals, and then invoke downstream learned skills for execution. However, the critical bottleneck lies in effectively bridging the gap between high-level semantic reasoning and low-level visuomotor execution. Existing solutions typically rely on either explicit kinematic states [ 20 ] or implicit multi-modal representations [ 8 ] . In practice, inferring explicit poses is highly vulnerable to perception noise in tight-clearance assembly, whereas utilizing goal images or latent embeddings poses a fundamental challenge for robots to physically ground them in the real world. In the context of interlocking brick assembly, this grounding challenge is exacerbated by two confounding factors: the visual ambiguity arising from periodic geometric units and repetitive visual features; and the spatial multi-modality of interlocking interfaces, where multiple locally valid yet discrete assembly poses exist. In light of these challenges, we explore a situated plan-to-execution interface to facilitate spatial grounding, alongside a modular formulation that encodes diverse assembly tasks into reusable primitives. To this end, we present BrickCraft , a framework that tackles long-horizon interlocking brick assembly by composing visuomotor skills with situated manual guidance. As illustrated in Fig. , given a 3D digital design, BrickCraft synthesizes a physically feasible assembly plan via an assembly-by-disassembly [ 23 , 13 ] approach. The primary innovation lies in anchoring each assembly step to a specific reference brick, a relative formulation that abstracts open-ended assembly tasks into a finite set of reusable primitive skills. To translate the symbolic plan into spatially grounded execution, we introduce situated manuals, a strategy that segments and highlights task-relevant entities directly within real-time robot observations to provide explicit visual guidance. Empirical evaluations demonstrate that situated manual guidance effectively resolves visual ambiguities and facilitates generalization to unseen brick structures, enabling the visuomotor skills to achieve an 86.25% success rate across 240 assembly trials on 48 distinct structures. Furthermore, these skills can be reliably composed to accomplish various long-horizon assembly tasks. Our main contributions are summarized as follows: • We present BrickCraft , a compositional framework for long-horizon interlocking brick assembly. • We propose a novel task formulation based on relative interlocking relationships that maps diverse assembly tasks to a finite set of primitive skills. • We propose situated manuals, a strategy that grounds symbolic intents into real-time robot observations to provide explicit spatial guidance for visuomotor execution. • We validate BrickCraft through comprehensive real-world experiments, demonstrating strong compositional generalization to unseen structures. II RELATED WORK II-A Robotic Assembly of Interlocking Bricks Interlocking brick assembly serves as an ideal long-horizon manipulation benchmark due to its universally accessible, standardized bricks and inherent combinatorial reconfigurability, where a limited set of elementary components can be assembled into diverse structures. Prior studies have approached robotic brick assembly through task and motion planning [ 9 ] , relying on carefully engineered setups to achieve the high-precision brick pose estimation and execution necessary for tight-clearance insertion [ 1 , 14 , 21 ] . Alternatively, reinforcement learning [ 7 ] has been explored to alleviate the reliance on precise spatial alignment, which remains restricted to isolated, single-step insertions. To achieve long-horizon assembly through data-driven approaches, ManualVLA [ 8 ] leverages pre-trained VLMs to infer intermediate subgoals that guide a downstream action expert. However, the reported execution horizon is relatively limited, and its scalability to more diverse and complex structures remains underexplored. In contrast, our work decomposes long-horizon assembly into reusable learned primitives, which enhances data efficiency and facilitates seamless adaptation to novel assembly tasks. II-B Visuomotor Imitation Learning Visuomotor imitation learning has emerged as a data-driven paradigm that learns to map raw visual observations to control actions by mimicking expert behaviors. Generative approaches like ACT [ 30 ] and diffusion policy [ 5 , 19 ] formulate policies as conditional action distributions over multimodal observations, showing remarkable efficacy in task-specific, fine-grained manipulation. Building upon these foundations, Vison-Language-Action (VLA) models [ 3 , 2 , 31 ] leverage large-scale multimodal pre-training to achieve broader semantic scene understanding and cross-task generalization. Despite these advancements, end-to-end policies often struggle with long-horizon reasoning and consistent multi-step execution. Recent Chain-of-Thought VLA frameworks [ 29 , 8 ] mitigate these challenges by introducing explicit reasoning chains to decompose complex tasks into interpretable sub-goals. Nevertheless, such reasoning often lacks physical grounding, neglecting fundamental assembly constraints such as collision avoidance and structural stability [ 26 , 22 , 12 ] , and exhibits limited proficiency in compositional generalization. II-C Skill Composition for Long-Horizon Manipulation Skill-centric frameworks achieve generalizable long-horizon manipulation by decomposing complex tasks into reusable primitive skills [ 28 , 20 , 24 , 9 , 13 ] , where each skill can either be a learned policy or a motion planning primitive. A high-level planner is then employed to query and compose these skills [ 15 , 11 ] . BrickCraft builds upon this paradigm to achieve long-horizon brick assembly. However, existing approaches typically guide the underlying policy using static skill embeddings [ 10 ] or fixed goal representations [ 20 ] . While effective for visually distinct objects, these representations are often semantically underspecified and perceptually ambiguous in the context of brick assembly. To address these

Read the complete text