Utilizing Inpainting for Keypoint Detection for Vision-Based Control of Robotic Manipulators

Sreejani Chatterjee \affilnum 1, Venkatesh Mullur \affilnum 1, Abhinav Gandhi \affilnum 1, and Berk Calli \affilnum 1 · 2026

Read Utilizing Inpainting for Keypoint Detection for Vision-Based Control of Robotic Manipulators by Sreejani Chatterjee \affilnum 1, Venkatesh Mullur \affilnum 1, Abhinav Gandhi \affilnum 1, and Berk Calli \affilnum 1 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13309.txt.

Introduction

Chatterjee and three colleagues build vision-based control for a robotic arm using only natural visual features. They attach ArUco markers to train a keypoint detector, use marker centres as labels, then inpaint the markers away to create markerless training images. At runtime a second inpainting model restores partly hidden robot regions, while an Unscented Kalman Filter steadies the estimates over time. The system controls the arm without a model under full and partial visibility, and extends to soft origami arms.

Catalog record

CREATORSreejani Chatterjee \affilnum 1, Venkatesh Mullur \affilnum 1, Abhinav Gandhi \affilnum 1, and Berk Calli \affilnum 1
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE106129
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.13309

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Utilizing Inpainting for Keypoint Detection for Vision-Based Control of Robotic Manipulators Sreejani Chatterjee \affilnum 1, Venkatesh Mullur \affilnum 1, Abhinav Gandhi \affilnum 1, and Berk Calli \affilnum 1 Abstract In this paper we present a novel visual servoing framework to control a robotic manipulator in the configuration space by using purely natural visual features. Our goal is to develop methods that can robustly detect and track natural features or keypoints on robotic manipulators that would be used for vision-based control, especially for scenarios where placing external markers on the robot is not feasible or preferred at runtime. For the model training process of our data driven approach, we create a data collection pipeline where we attach ArUco markers along the robot’s body, label their centers as keypoints, and then utilize an inpainting method to remove the markers and reconstruct the occluded regions. By doing so, we generate natural (markerless) robot images that are automatically labeled with the marker locations. These images are used to train a keypoint detection algorithm, which is used to control the robot configuration using natural features of the robot. Unlike the prior methods that rely on accurate camera calibration and robot models for labeling training images, our approach eliminates these dependencies through inpainting. To achieve robust keypoint detection even in the presence of occlusion, we introduce a second inpainting model, this time to utilize during runtime, that reconstructs occluded regions of the robot in real time, enabling continuous keypoint detection. To further enhance the consistency and robustness of keypoint predictions, we integrate an Unscented Kalman Filter (UKF) that refines the keypoint estimates over time, adding to stable and reliable control performance. We obtained successful control results with this model-free and purely vision-based control strategy, utilizing natural robot features in the runtime, both under full visibility and partial occlusion. keywords Image inpainting, Vision-based control, keypoint detection † † runninghead: Chatterjee et al. † † affiliation: \affilnum 1Robotics Engineering Program, Worcester Polytechnic Institute, MA, USA † † corresponding: Sreejani Chatterjee, Worcester Polytechnic Institute, 100 Institute Rd, Worcester, MA 01609, USA. 1 Introduction Figure 1 : Image (a) shows the original images for different robot configurations covering the workspace with ArUco markers attached, and image (b) presents the corresponding binary masks highlighting these markers. Both (a) and (b) are provided as input to a modified LaMa inpainting model, which generates the corresponding reconstructed images shown in (c). The reconstructed images, along with the original ArUco marker centers, are then used to generate a labeled training dataset, as shown in image (d), where the green bounding boxes are drawn with the same orientation of the ArUco markers and the red dots are the marker centers. This dataset is used to train a modified keypointrcnn_resnet50_fpn model for real-time keypoint prediction. Image (e) illustrates the model’s prediction for an arbitrary configuration. (f) depicts the start and goal configurations, along with the resulting control trajectory generated using the predicted keypoints. (g–l) depicts the same control experiment as (f) with occlusion in the environment. (g), (i), (k) show occluded start, intermediate, and goal frames; (h), (j), (l) are their inpainted counterparts. (m) depicts predicted control trajectory under occlusion, corresponding to (f), using inpainted keypoints. Figure 2 : The data collection phase for out-of-plane motion. Image (a) shows the original images for different robot configurations covering the workspace in 3D with ArUco markers attached, and image (b) presents the corresponding binary masks highlighting these markers. Both (a) and (b) are provided as input to a modified LaMa inpainting model, which generates the corresponding reconstructed images shown in (c). The reconstructed images, along with the original ArUco marker centers, are then used to generate a labeled training dataset, as shown in image (d), where the green bounding boxes are drawn with the same orientation of the ArUco markers and the red dots are the marker centers. This dataset is used to train a modified keypointrcnn_resnet50_fpn model for real-time keypoint prediction. Image (e) illustrates the model’s prediction for an arbitrary out-of-plane configuration. (f) depicts the start and goal configurations, along with the resulting control trajectory generated using the predicted keypoints. Vision-based control methods [ Hutchinson et al.(1996) , Hashimoto(2003) ] enable robotic manipulators to operate effectively in unstructured or dynamic environments by utilizing task-relevant visual feedback to close the control loop. These approaches improve robustness to model inaccuracies making them ideal to control systems with complex or variable dynamics [ Calli and Dollar(2016) , Cuevas-Velasquez et al.(2018) , Ardón et al.(2018) ] such as soft or continuum robots. Model-free visual servoing further minimizes the reliance on predefined robot models [ Wang et al.(2018) , Navarro-Alarcon and Liu(2017) ] by learning robot-feature motion model during control. This technique is particularly advantageous for robots that are difficult to model accurately, such as soft robots [ Lai et al.(2020) , Luo et al.(2018) ] , underactuated systems [ Liu et al.(2020) , Gandhi et al.(2023) ] , 3D-printed platforms [ Chavdarov et al.(2019) , Onal et al.(2014) ] , and robots built with low-cost hardware [ Adzeman et al.(2020) ] . Our work focuses on image-based visual servoing (IBVS) in an eye-to-hand configuration [ Tokuda et al.(2021) ] , where visual features are extracted from a fixed external camera observing the robot. Mostly in literature, eye-to-hand visual servoing implementations have primarily relied on feature-rich patterns or fiducial markers attached to the robot’s end effector [ Lai et al.(2020) , Calli and Dollar(2016) , Cuevas-Velasquez et al.(2018) , Ardón et al.(2018) ] and/or along the body of the robot [ Gandhi et al.(2022) ] . If the markers are occluded possibly due to self-occlusion or other objects in the scene [ Lippiello et al.(2005) ] , or remain undetected due to motion blur, reflections or lighting conditions [ Mondéjar-Guerra et al.(2018) ] , the control operation may fail. Also, utilizing markers may be inconvenient, impractical, or infeasible in various applications. Our research goal is to advance the limits of image-based visual servoing by removing the reliance on these external markers by using purely natural features along the robot’s body in image space, without attaching any external markers. While doing so, we also strive to reduce the dependencies on explicit robot or camera models or proprioceptive sensors on the robot. To achieve this we took inspiration from the success of human pose estimation in computer vision for robust keypoint detection in image without using external markers. These algorithms detect the location of specific keypoints corresponding to human body parts, e.g. hands, shoulders, eyes in an image. There are ample datasets available in the literature where these body parts are annotated as keypoints such as MPII Human Pose [ Andriluka et al.(2014) ] , COCO test-dev [ Lin et al.(2014) ] , Densepose-COCO [ Güler et al.(2018) ] . These datasets have enabled the training of deep learning frameworks such as OpenPose [ Cao et al.(2021) ] and DeepPose [ Toshev and Szegedy(2014) ] , which can reliably detect and track human body keypoints in images captured under unconstrained, real-world conditions. Despite their proven success in human pose estimation, similar keypoint datasets for robotic manipulators are sparse in the literature. In [ Lee et al.(2020) ] the authors create datasets of three rigid

Read the complete text