Developing Modular Grasping and Manipulation Pipeline Infrastructure to Streamline Performance Benchmarking

Brian Flynn, Kostas Bekris, Berk Calli, Aaron Dollar, Adam Norton, Yu Sun, Holly Yanco · 2025

Read Developing Modular Grasping and Manipulation Pipeline Infrastructure to Streamline Performance Benchmarking by Brian Flynn, Kostas Bekris, Berk Calli, Aaron Dollar, Adam Norton, Yu Sun, Holly Yanco (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-06819.txt.

Introduction

Brian Flynn led a team building shared infrastructure for robot grasping research in a 2025 paper. Robot manipulation groups struggle to combine open-source parts and reproduce each other's results. That makes fair comparison of grasping methods hard, since groups rely on whole-system tests that hide where gains come from. Their project builds a modular pipeline where pieces can be swapped in a standard way.

The paper gives a high-level view of the pipeline's design. It also reports experiments run while exercising the system during development. Planned work will widen its modularity. The wider aim is shared standards and guidelines for open development and benchmarking in robot manipulation.

Catalog record

CREATORBrian Flynn, Kostas Bekris, Berk Calli, Aaron Dollar, Adam Norton, Yu Sun, Holly Yanco
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE10 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-06819

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Developing Modular Grasping and Manipulation Pipeline Infrastructure to Streamline Performance Benchmarking Brian Flynn, Kostas Bekris, Berk Calli, Aaron Dollar, Adam Norton, Yu Sun, Holly Yanco Abstract I Introduction and Background Based on our work engaging with the robot manipulation community through online surveys, holding conference workshops, and deep dive interviews with community stakeholders, we have determined several limitations of the current open-source and benchmarking ecosystem with recommendations for improvement. Current limitations include issues integrating open-source products and a lack of truly modular software to enable component-level and holistic system evaluations. This aligns with previously published sentiments around issues with code reproducibility [ 1 , 2 ] and defining benchmarks that evaluate components of a manipulation solution, such as the Cluttered Environment Picking Benchmark [ 3 ] . Recommendations for improvement include organized repositories of open-source products and benchmarking assets, and the benefit of modular software components was highlighted. Projects like SceneReplica [ 4 ] demonstrate the effectiveness of benchmarking with interchangeable pipeline components (perception, grasp planning, motion planning, and control), and the GRASPA project [ 5 ] provides an example implementation on the Franka Panda robot to benchmark the performance of three grasp planners. With these factors in mind, we are developing a modular grasping and manipulation pipeline infrastructure for improved performance benchmarking. As part of the COMPARE Ecosystem project [ 6 ] , we intend to use this infrastructure to establish standards and guidelines towards improving open-source development and benchmarking. This paper presents a brief review of experiments conducted during development of the pipeline infrastructure, the advancements enabled by each experiment, an overview of the current architecture of the pipeline infrastructure, and future work. II Pipeline Infrastructure The goal of the pipeline infrastructure is to dictate the flow of an experiment using state machines with nested behavior trees, wherein users will utilize existing ROS packages or create interchangeable components (e.g., grasp planning, motion planning, perception) following established guidelines to “drop in.” Components should be interchangeable to be readily swapped out without having to write new code (e.g., each grasp planning service utilizes the same request and response message type such as image in, pose out) and the user should make minimal modifications to the infrastructure set-up to match their testing needs. The infrastructure is also intended to be sufficiently hardware-agnostic (i.e., functional regardless of hardware, assuming that the hardware is ROS compatible and a ROS driver exists for such hardware). It is likely that ROS1 is still the preferred implementation for many research groups given that ROS2 is still largely in development, so streamlined simulation environment integrations and certain robot drivers may not yet exist. The community is expected to shift to ROS2 at large due to the fact that ROS1 is nearing end-of-life, so we are converting the codebase to ROS2 despite the fact that the pipeline infrastructure was initially developed and tested using ROS1 assets. Various components of the pipeline infrastructure are in development and can be found on our organization Github page 1 1 1 https://github.com/uml-robotics (e.g., components for point cloud processing 2 2 2 https://github.com/uml-robotics/robot-common-3d ). Our prior pipeline infrastructure development in ROS1 was built around FlexBe 3 3 3 http://philserver.bplaced.net/fbe/ as a state-machine-driven manipulation process. It used concise states that make requests to ROS services and actionservers to execute certain tasks (e.g., motion planning via MoveIt, filtering point clouds via PCL) that could be easily added, removed, or reordered as needed using the FlexBe user interface or directly through code; these states are wrapped into a process container called a behavior. The manipulation utilities we developed 4 4 4 https://github.com/uml-robotics/armada_behaviors were designed to be robot-agnostic and typically relied on the ROS parameter server to handle varying hardware requirements. While FlexBe is undergoing development for ROS2 environments 5 5 5 https://github.com/FlexBE/flexbe_behavior_engine , it was built upon a python state-machine library called SMACH 6 6 6 http://wiki.ros.org/smach which is not inherently dependent on ROS1 or ROS2. The SMACH library can be used to provide nearly identical functionality albeit without the aid of the FlexBe UI which, while helpful, is not a necessary component. This allows us to continue development on the pipeline infrastructure in ROS2 without relying on the completion of certain external resources. III Experimentation Fig. 1: Experiments performed during development. During development of the pipeline infrastructure, multiple experiments were conducted to exercise the infrastructure in our labs at UML and WPI as well as our partners’ labs at OSU (see Fig. 1 ). All of these experiments led to further improvements to the modularity of the infrastructure to allow for more variety of components that could be easily implemented within the pipeline. They also validated the functionality of the pipeline to work with code contributed by multiple internal and external users. Experiment 1 : Performance of the Grasp Pose Detection (GPD) [ 7 ] algorithm was evaluated by deploying it within the initial FlexBe manipulation pipeline infrastructure across two robot embodiments (UR5e and Kinova Gen3 with the same gripper and sensor) at UML. Following the grasp planning benchmarking protocol in Bekiroglu et al. [ 8 ] , 640 grasps were performed on two objects under two lighting conditions and with or without 10 cm workspace elevation (unpublished). Each individual robot, gripper, and sensor embodiment was controlled using the same FlexBe behavior with the only change being a set of parameters which handled lower-level differences between systems. Experiment 2 : State machine functionality was expanded to control three physical test apparatuses that automatically reset task conditions (object repositioning after grasping, door and drawer closing after opening) to enable automated, repeated testing for multiple trials. The manipulation behavior was nested inside of a test apparatus reset behavior which controlled data recording and automated resetting of apparatuses between manipulation operations. Using a Kinova Gen3, our partners at OSU performed 1,020 grasps on four objects using the grasp reset mechanism [ 9 ] , and 600 grasps using a Kinova Jaco2 were performed on the door and drawer reset mechanisms [ 10 ] . Experiment 3 : The modularity of the pipeline was further expanded by integrating four vision-based grasping algorithms (GG-CNN, ResNet, Top Surface, and Mask-Based). These algorithms were originally used in a pipeline developed at WPI on a Franka Emika Panda with the Panda gripper. The grasping algorithms were compatible with the FlexBe pipeline architecture because they used ROS service interactions; requesting a sensor image and returning a pose or list of poses, allowing them to effectively swap places with the grasping algorithm initially used within the infrastructure. Using ten objects against textured or non-textured backgrounds, two lighting conditions, and again following the grasp planning benchmarking protocol in Bekiroglu et al. [ 8 ] , 5,040 grasps were performed and their performance compared [ 11 ] . A Docker set-up to reproduce this benchmarking is available on GitHub 7 7 7 https://github.com/vinayakkapoor/vision_based_grasping_benchmarking . 8 8 8 https://github.com/moveit/moveit_grasps IV Future Work Grasp Planner Input Data Output Pose Dex-Net (GQ-CNN) [ 12 ] Depth image 2D grasp rectangle (x, y, width, height, angle) GQ-CNN [ 13 ] Depth image 2D grasp rectangle (x, y, width, height, angle) with grasp success probability GPD [ 7 ] Point cloud 6-DoF grasp pose (x, y, z, r, p, y) PointNetGPD [

Read the complete text