AdaptPNP: Integrating Prehensile and Non-Prehensile Skills for Adaptive Robotic Manipulation

Jinxuan Zhu, Chenrui Tie, Xinyi Cao, Yuran Wang, Jingxiang Guo, Zixuan Chen, Haonan Chen, Junting Chen, Yangyu Xiao, Ruihai Wu, Lin Shao · 2025

Read AdaptPNP: Integrating Prehensile and Non-Prehensile Skills for Adaptive Robotic Manipulation by Jinxuan Zhu, Chenrui Tie, Xinyi Cao, Yuran Wang, Jingxiang Guo, Zixuan Chen, Haonan Chen, Junting Chen, Yangyu Xiao, Ruihai Wu, Lin Shao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-11052.txt.

Introduction

Jinxuan Zhu, Chenrui Tie, and their colleagues present AdaptPNP, a planning framework that lets robots mix grasping moves with pushes, pokes, and slides. Many tasks cannot be done by grasping alone, yet a robot must decide when a push is better, which move fits, and how to chain moves into a sturdy plan. That joint choice across tasks and settings is the core challenge.

A vision-language model reads the scene and the task text and drafts a high-level plan that orders grasp and non-grasp moves. A digital twin layer predicts where objects should end up, so the robot can rehearse in its head. A control layer then turns the plan into motion.

Catalog record

CREATORJinxuan Zhu, Chenrui Tie, Xinyi Cao, Yuran Wang, Jingxiang Guo, Zixuan Chen, Haonan Chen, Junting Chen, Yangyu Xiao, Ruihai Wu, Lin Shao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-11052

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AdaptPNP: Integrating Prehensile and Non-Prehensile Skills for Adaptive Robotic Manipulation Thanks: * Equal Contribution Jinxuan Zhu, Chenrui Tie, Xinyi Cao, Yuran Wang, Jingxiang Guo, Zixuan Chen, Haonan Chen, Junting Chen, Yangyu Xiao, Ruihai Wu, Lin Shao Abstract Non-prehensile (NP) manipulation, in which robots alter object states without forming stable grasps (for example, pushing, poking, or sliding), significantly broadens robotic manipulation capabilities when grasping is infeasible or insufficient. However, enabling a unified framework that generalizes across different tasks, objects, and environments while seamlessly integrating non-prehensile and prehensile (P) actions remains challenging: robots must determine when to invoke NP skills, select the appropriate primitive for each context, and compose P and NP strategies into robust, multi-step plans. We introduce AdaptPNP, a vision-language model (VLM)-empowered task and motion planning framework that systematically selects and combines P and NP skills to accomplish diverse manipulation objectives. Our approach leverages a VLM to interpret visual scene observations and textual task descriptions, generating a high-level plan skeleton that prescribes the sequence and coordination of P and NP actions. A digital-twin based object-centric intermediate layer predicts desired object poses, enabling proactive mental rehearsal of manipulation sequences. Finally, a control module synthesizes low-level robot commands, with continuous execution feedback enabling online task plan refinement and adaptive replanning through the VLM. We evaluate AdaptPNP across representative P&NP hybrid manipulation tasks in both simulation and real-world environments. These results underscore the potential of hybrid P&NP manipulation as a crucial step toward general-purpose, human-level robotic manipulation capabilities. Project Website: https://sites.google.com/view/adaptpnp/home I Introduction When manipulating objects to achieve desired configurations, robots typically rely on establishing stable grasps and transporting objects to target locations. However, prehensile manipulation often becomes infeasible or insufficient due to geometric constraints, potential collisions, or object characteristics that preclude stable grasping. For instance, when transferring a thin card lying flat on a tabletop that cannot be grasped directly, a robot could first push it toward the table edge, then grasp and lift it from the side. Such scenarios necessitate non-prehensile (NP) manipulation strategies [ 34 ] , where robots alter object states through contact interactions like pushing, poking, or sliding without forming stable grasps. The integration of prehensile and non-prehensile skills significantly expands robotic manipulation capabilities, enabling more versatile and human-like object interactions. However, effectively incorporating non-prehensile skills into robotic manipulation frameworks presents significant challenges. First, the diversity of NP primitives ( e.g. , pushing, poking, pressing, sliding) dramatically increases manipulation dexterity but also creates combinatorial complexity in skill selection and sequencing. Second, robots must reason about and compose P&NP actions based on environmental constraints, object properties, and task requirements, which demands sophisticated understanding of skill preconditions, effects, and inter-dependencies. Third, this combinatorial explosion is compounded by the inherently multi-modal nature of manipulation tasks: numerous skill combinations may appear semantically reasonable, yet only a subset proves physically feasible or execution-efficient under physical constraints. Together, these factors complicate the generation of robust, executable task plans that balance semantic plausibility, physical feasibility, and execution reliability. Fig. 1 : Overview of AdaptPNP. A VLM-based task planner generates a mixed sequence of prehensile (grasp, moveto, release) and non-prehensile (push, rotate) primitives; a digital twin “mentally rehearses” each primitive by generating target 6D object poses; and a closed-loop reflection mechanism uses execution feedback to iteratively refine the plan. Existing methods often lack adaptive support for integrating prehensile and non-prehensile skills. Single-task NP policies [ 6 , 41 , 8 , 22 , 37 , 27 , 24 ] excel at isolated primitives like pushing or sliding, but cannot choose among multiple NP options or integrate grasping based on scene context. Classical task and motion planning (TAMP) frameworks [ 1 , 12 , 45 ] sequence diverse actions using symbolic models ( e.g. , PDDL [ 1 ] ) rely on extensive hand-crafted domain knowledge, limiting adaptability to novel objects and environments. VLM-based planners represent tasks as natural-language subgoals [ 25 ] , image-space waypoints [ 47 ] or spatial relations between 3D points [ 15 ] , which cannot fully specify NP action in all spatial directions ( e.g. , poking to induce rotation) and often omit closed-loop plan refinement. A general P&NP framework must adaptively integrate P&NP skill according to visual and physical features of task, object and environment. To address these challenges, we present AdaptPNP, an integrated task-and-motion planning framework that unifies high-level P&NP skill scheduling with low-level execution. At its core, a VLM takes an RGB scene and natural-language task instruction to infer a sequence of prehensile and non-prehensile primitives, leveraging commonsense knowledge for zero-shot generalization to novel objects and environments. For each primitive, multiple candidate 6DoF subtarget object poses are generated according to task instruction and filtered through a digital twin to discard physically infeasible options, providing precise intermediate objectives for NP actions. Selected subtargets feed into off-the-shelf motion planning module to execute the corresponding motions. Crucially, after executing each skill, visual observations and controller feedback provide physical insights to the VLM task planner, enabling it to identify physically feasible skill sequences among multiple semantically plausible alternatives and resolve multi-modal ambiguities in P&NP planning. We evaluate AdaptPNP on eight challenging P&NP hybrid manipulation tasks in both simulation and real-world settings. AdaptPNP consistently outperforms non-hierarchical reinforcement-learning and MPC approaches, hierarchical VLM-based planning methods, and end-to-end vision-language-action models. These results demonstrate that AdaptPNP exactly expands robotic capabilities and effectively handles diverse, complex manipulation challenges. To summarize, our key contributions are: • We propose AdaptPNP, a generalizable manipulation framework that adaptively selects and coordinates prehensile and non-prehensile skills based on task requirements, object properties, and environmental constraints, enabling flexible skill composition across diverse manipulation scenarios through VLM-guided planning. • We introduce a digital-twin based intermediate representation that generates physically-informed 6D object target poses of each skill, bridging the planning-execution gap by capturing both translational and rotational object states while respecting physical constraints. • We demonstrate comprehensive evaluation across diverse manipulation scenarios requiring coordinated P&NP skills, including object alignment, extrinsic dexterity exploitation, and tool using, achieving superior performance over existing approaches. II related work II-A Non-Prehensile Manipulation Skills Non-prehensile manipulation tasks can be broadly categorized into three types. First, contact-based non-grasping manipulation directly changes object states via interactions like pushing [ 8 , 41 , 51 , 19 ] , poking [ 53 , 17 , 6 , 28 , 52 ] , or sliding [ 5 , 46 ] , in which robots apply contact forces and friction without ever forming a stable grasp to achieve the desired motion; Second, extrinsic dexterity leverages environmental features, such as walls, edges, slots, or slopes, to enable complex manipulations [ 8 , 41 , 39 , 20 , 3 ] ; Third, tool-mediated manipulation employs auxiliary tools like hooks [ 22 , 36 ] , push bars [ 11 ] , or rods and hammers [ 18 , 37 , 14 ] to extend the robot’s effective reach and manipulation capabilities. While these approaches demonstrate the effectiveness of individual NP skills, they typically rely on task-specific action pattern and policy, limiting their adaptability to novel tasks and scenarios. II-B Task and Motion Planning Task and motion planning (TAMP) seeks to jointly reason over high‐level symbolic actions and low‐level continuous motions to generate executable plans for robotic systems.

Read the complete text