PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation

Yifan Yin Zhengtao Han Shivam Aarya Jianxin Wang Shuhang Xu Jiawei Peng, [0.05in] Angtian Wang Alan Yuille Tianmin Shu, [0.05in] Johns Hopkins University ShanghaiTech University[0.05in] https://partinstruct.github.io · 2025

Read PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation by Yifan Yin Zhengtao Han Shivam Aarya Jianxin Wang Shuhang Xu Jiawei Peng, [0.05in] Angtian Wang Alan Yuille Tianmin Shu, [0.05in] Johns Hopkins University ShanghaiTech University[0.05in] https://partinstruct.github.io (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21652.txt.

Introduction

Yifan Yin and co-authors build a benchmark for robots that must handle parts of objects, not just whole ones. Turning a bottle to show its label demands part-level understanding that today's instruction-following robots rarely get tested on. PartInstruct supplies 513 objects in 14 categories, each labeled by parts, and 1,302 fine-grained tasks in 16 classes.

Over 10,000 expert demonstrations, synthesized in a 3D simulator, pair each task with a plain instruction, a chain of part-based skill steps, and exact 3D ground truth. A test suite probes new positions, objects, and tasks. Leading methods, both end-to-end learners and two-level planners, struggle to ground part concepts and to sustain long part-focused jobs, marking clear headroom.

Catalog record

CREATORYifan Yin Zhengtao Han Shivam Aarya Jianxin Wang Shuhang Xu Jiawei Peng, [0.05in] Angtian Wang Alan Yuille Tianmin Shu, [0.05in] Johns Hopkins University ShanghaiTech University[0.05in] https://partinstruct.github.io
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-21652

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation Yifan Yin Zhengtao Han Shivam Aarya Jianxin Wang Shuhang Xu Jiawei Peng, [0.05in] Angtian Wang Alan Yuille Tianmin Shu, [0.05in] Johns Hopkins University ShanghaiTech University[0.05in] https://partinstruct.github.io Abstract Fine-grained robot manipulation, such as lifting and rotating a bottle to display the label on the cap, requires robust reasoning about object parts and their relationships with intended tasks. Despite recent advances in training general-purpose robot manipulation policies guided by language instructions, there is a notable lack of large-scale datasets for fine-grained manipulation tasks with part-level instructions and diverse 3D object instances annotated with part-level labels. In this work, we introduce PartInstruct, the first large-scale benchmark for both training and evaluating fine-grained robot manipulation models using part-level instructions. PartInstruct comprises 513 object instances across 14 categories, each annotated with part-level information, and 1302 fine-grained manipulation tasks organized into 16 task classes. Our training set consists of over 10,000 expert demonstrations synthesized in a 3D simulator, where each demonstration is paired with a high-level task instruction, a chain of base part-based skill instructions, and ground-truth 3D information about the object and its parts. Additionally, we designed a comprehensive test suite to evaluate the generalizability of learned policies across new states, objects, and tasks. We evaluated several state-of-the-art robot manipulation approaches, including end-to-end vision-language policy learning and bi-level planning models for robot manipulation on our benchmark. The experimental results reveal that current models struggle to robustly ground part concepts and predict actions in 3D space, and face challenges when manipulating object parts in long-horizon tasks. An example fine-grained robot manipulation task in PartInstruct . To successfully perform the task described in the instruction (e.g., showing the cap without occluding it), the robot needs to reason about what object parts are relevant, ground the parts to its 3D visual perception, and plan for a sequence of part-level manipulation skills (e.g., the bottom sequence). Native object manipulation without a detailed understanding of object parts will fail to achieve the intended goal (e.g., the top sequence). These tasks thus pose challenges for robust 3D vision and part-level grounding and reasoning. We show more examples on the project website. 1 1 footnotetext: Equal contribution. Zhengtao Han completed this work during an internship at JHU. I Introduction There has been an increasing interest in training general-purpose vision-language policies for robot manipulation guided by language instructions [ 24 , 17 , 50 , 16 , 32 , 51 ] , particularly with the recent advances in large generative models [ 3 , 41 ] . These models represent a promising type of method for solving general robot manipulation problems, as they have the potential to follow natural language instructions to complete any described task. Prior works on language-guided robot manipulation have been mainly focused on high-level manipulation tasks involving simple objects (such as rearranging blocks). However, in the real world, robots often need to perform fine-grained manipulation of diverse everyday objects, in which the robots need to not only identify the target object but also understand and interact with specific parts of that object to perform the intended task as instructed. This involves reasoning about the relationship between the part and the task, and grounding that understanding into precise motion planning. For instance, to successfully perform the manipulation task defined in the instruction as shown in Figure PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation , the robot needs to identify crucial parts of the object relevant to the task (e.g., the label on the cap of the bottle) and reason about a chain of base part-based skills that would lead to the desired goal state implied by the instruction, which is to display the label clearly to the human user without occlusion. Table I: Comparison of PartInstruct with existing tabletop robot manipulation benchmarks based on: the number of distinctive part-level instructions, the number of part labels, the number of fine-grained part-level tasks, availability of training demonstrations, and whether these demonstrations include part-level annotations such as 2D and 3D segmentation masks. Name # Part Instruct # Part Labels # Part-level Tasks Demo 2D Part Mask 3D Part Mask CALVIN 6 - 6 ✓ ✗ ✗ RLbench 136 - 64 ✗ ✗ ✗ VIMAbench - - - ✓ ✗ ✗ LoHoRavens - - - ✓ ✗ ✗ ManiSkill (SAPIEN) - 14,068 - ✓ ✗ ✗ PartManip - 8,489 1,432 ✗ ✗ ✗ Open6DOR 2,447 - 1,419 ✗ ✗ ✗ PartInstruct (ours) 4,043 4,653 1,302 ✓ ✓ ✓ Despite the importance of part-level perception and reasoning for robot manipulation, existing robot manipulation benchmarks on instruction following lack comprehensive integration of part-level semantics in both task instructions and object ground-truth annotations [ 16 , 17 , 24 , 46 , 50 , e.g.,] . These benchmarks focus on object instance-level manipulation tasks but do not include fine-grained, part-level manipulation tasks like the example in Figure PartInstruct: Part-level Instruction Following for Fine-grained Robot Manipulation . There have been recent benchmarks that evaluate fine-grained, part-level manipulation tasks, but they either lack language instructions [ 27 , 9 , e.g.,] or do not provide training data for policy learning [ 7 , e.g.,] . To address these gaps, we introduce PartInstruct , the first large-scale fine-grained robot manipulation benchmark for vision-language policy learning that incorporates part-level semantics. Our core idea is to develop part-level skills that enable robots to perform complex, fine-grained object manipulation tasks, including those requiring long-horizon motion plans. We developed a robot manipulation simulator for part-level instruction following tasks, PartGym . Built upon the PartGym simulator, our PartInstruct benchmark supports both training and evaluation models on part-level manipulation tasks. Specifically, we provide a large set of 3D assets of everyday objects richly annotated with part-level information. Using these object assets and detailed annotations, we created a large-scale training dataset of expert demonstrations. Each demonstration is paired with a task instruction as well as a chain of base skill instructions (such as touching or grasping an object part) necessary for performing the overall task. This dataset allows training models for both long-horizon manipulations guided by task instructions for long-horizon planning and base manipulation skills guided by skill instructions. Additionally, we developed a comprehensive evaluation suite consisting of five test sets, each corresponding to a different type of generalization test. Together, these tests assess how well a learned policy performs in unseen scenarios, including new states, objects, and tasks. We compare PartInstruct with several existing table-top manipulation benchmarks in Table I . We evaluated multiple state-of-the-art vision-language policy learning methods designed for language-guided robot manipulation. We also combined recent learning-based low-level action policy planning models and VLM-based high-level task planners to create strong bi-level planning baselines for fine-grained manipulation tasks, which explicitly reason about object parts relevant to a task and how to interact with them to achieve the final goal. Our experimental results demonstrate that state-of-the-art methods still struggle with complex fine-grained manipulation tasks. We also show that visual representations based on robust part-level 3D perception can significantly improve model performance. These results help reveal the fundamental building blocks for fine-grained task manipulation. In summary, our main contribution includes (1) the first part-level instruction following benchmark for both training and evaluating fine-grained robot manipulation models’ capacity for

Read the complete text