DIJE: Dense Image Jacobian Estimation for Robust Robotic Self-Recognition and Visual Servoing

Yasunori Toshimitsu, Kento Kawaharazuka, Akihiro Miki, Kei Okada, Masayuki Inaba · 2025

Read DIJE: Dense Image Jacobian Estimation for Robust Robotic Self-Recognition and Visual Servoing by Yasunori Toshimitsu, Kento Kawaharazuka, Akihiro Miki, Kei Okada, Masayuki Inaba (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-00446.txt.

Introduction

Yasunori Toshimitsu and four co-authors teach a robot to tell its own motion from motion caused by others, in a research paper from 2025. To move safely in the real world, a robot must know the state of its own body and of tools it holds. Their method needs no markers and no prior model of the robot.

The method estimates, for every pixel in the camera view, how the image will shift when the robot moves. It runs fast enough for live use on the full image. Tested on a physical muscle-driven robot, it let the robot tell self-motion from outside motion. A controller built on it learned reaching and two-handed tool tasks.

Catalog record

CREATORYasunori Toshimitsu, Kento Kawaharazuka, Akihiro Miki, Kei Okada, Masayuki Inaba
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-00446

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DIJE: Dense Image Jacobian Estimation for Robust Robotic Self-Recognition and Visual Servoing Yasunori Toshimitsu, Kento Kawaharazuka, Akihiro Miki, Kei Okada, Masayuki Inaba Abstract For robots to move in the real world, they must first correctly understand the state of its own body and the tools that it holds. In this research, we propose DIJE, an algorithm to estimate the image Jacobian for every pixel. It is based on an optical flow calculation and a simplified Kalman Filter that can be efficiently run on the whole image in real time. It does not rely on markers nor knowledge of the robotic structure. We use the DIJE in a self-recognition process which can robustly distinguish between movement by the robot and by external entities, even when the motion overlaps. We also propose a visual servoing controller based on DIJE, which can learn to control the robot’s body to conduct reaching movements or bimanual tool-tip control. The proposed algorithms were implemented on a physical musculoskeletal robot and its performance was verified. We believe that such global estimation of the visuomotor policy has the potential to be extended into a more general framework for manipulation. I Introduction While robots have predominantly been used in factory environments for decades, they have only recently started to be used in everyday environments, which can be unstructured and unpredictable. These uncertainties exist both in the external environment and the robot’s own body. In particular, we consider that it is important for the robot to be able to discover its own body while actively moving within the environment. We can consider the problem of visual self-body control as a two component problem. For one, the robot must be able to recognize the extent of its own body , i.e. a self-recognition problem. It must be generalize to recognize novel visual features such as new tools in its hands, while ignoring external movements not caused by the robot. Also, the robot must recognize how to control its own body . This is akin to acquiring an internal model of the robot movement usable in control, a simple example being the manipulator Jacobian matrix. To achieve this, a holistic understanding of the visuomotor system is required, rather than calculating values discretely for predetermined keypoints or markers along the robot. To that end, we propose DIJE (Dense Image Jacobian Estimator), a method to densely (i.e. for every pixel of the image) estimate the image Jacobian based on exteroceptive camera images and proprioceptive joint sensor data. It does not require a priori knowledge of the robot’s kinematic structure, and the resulting dense image Jacobian encodes the global relationship between visual features and joint information for the whole image. Multiple image Jacobian estimation methods have been proposed [ 1 , 2 , 3 , 4 , 5 ] , but to our knowledge, this work is the first to apply it densely on the image. To enable this, we propose a simplified Kalman Filter-based estimation algorithm for the image Jacobian, and an update rule to shift the image Jacobian estimation value in each timestep based on the robot movement. DIJE enables a unified approach to self-recognition and visual servoing control. We propose using the output of DIJE to a self-recognition algorithm which can robustly predict the extent of the robot’s own body even in the presence of external movements, and to a markerless visual servoing controller which can learn to control the robot’s body and the tools in its hands. The contributions of this work are as follows: • a dense estimation method for the image Jacobian implemented by a recursive algorithm, based on the Kalman Filter and a dense update method across timesteps, robust to non robot-induced external movements • a dense binary labeling algorithm of the robot’s self body based on the output of DIJE, capable of distinguishing between self-induced and external movement • a markerless visual servoing controller based on the output of DIJE All of the algorithms presented require no prior knowledge of the robot structure, and can be run in real time on a conventional notebook PC. II Related Work II-A Robotic Self Recognition In a robotic self recognition system, the robot attempts to locate and distinguish its own body from the environment in the visual (or depth) image. They can be split into methods requiring prior knowledge of the robotic structure [ 6 , 7 , 8 , 9 ] , and those that don’t [ 10 , 11 , 12 ] . For methods requiring a priori knowledge, the recognition algorithm is trained based on datasets synthesized from a simulation of the robot. Machine learning algorithms such as Mask R-CNN [ 6 ] , CNN [ 7 ] , or random forest [ 8 , 9 ] has been used for the recognizer. In the simulator, perfect knowledge of the robot state is available. These recognition models can take advantage of that fact to estimate other features from the image and not just the location or binary mask of the robot body, such as joint angle pose [ 7 , 8 , 9 ] . However, since a training process is required, the application of these methods to recognition of novel objects attached to the robot (such as tools) is not possible. Model-less methods for self-recognition usually utilize the motion of the robot to segment its body from the background. In the work by Metta and Fitzpatrick, the optical flow is used to detect motion, and as its direction changes depending on joint movement, it is compared to the joint angle measurement to detect the location of the arm [ 10 ] . Other methods similarly use the detected motion and correlate it with the joint movement, by calculating the mutual information [ 12 ] or comparing the temporal correlation [ 11 ] to determine whether to attribute it to the robot. II-B Uncalibrated Visual Servoing In uncalibrated visual servoing, the manipulator model and camera parameters are not known a priori and must be learned online while the robot moves. The image Jacobian J describes the differential relationship between a point in the image of the robot’s body and the robot’s joint pose. \dot{\bm{p}}=\bm{u}=J(\bm{q})\dot{\bm{q}} (1) where \bm{p} is the time-varying image coordinate of a point on the robot, \bm{u} is its time derivative, i.e. the optical flow, and \dot{\bm{q}} is the joint velocity. As the pseudoinverse of the image Jacobian can be used to control the image coordinates of the robot, the online estimation of J has been intensely researched for uncalibrated visual servoing, with methods being proposed based on Broyden updating [ 1 ] , Kalman Filtering [ 5 , 3 ] , support vector regression [ 2 ] , or least-squares fitting [ 4 ] . In these works, the estimated image Jacobian has been successfully applied to visual servoing to control robots whose kinematic structures are unknown. However, many of the uncalibrated visual servoing methods assume that the point of interest being controlled is known a priori , for example by markers [ 1 , 5 ] or is only verified on a simulated model [ 4 , 3 , 2 ] where perfect knowledge is available for where each point of the robot

Read the complete text