6D Object Pose Tracking in Internet Videos for Robotic Manipulation
Georgy Ponimatkin Martin Cífka Tomáš Souček Médéric Fourmy, Yann Labbé Vladimír Petrík Josef Sivic · 2025
Read 6D Object Pose Tracking in Internet Videos for Robotic Manipulation by Georgy Ponimatkin Martin Cífka Tomáš Souček Médéric Fourmy, Yann Labbé Vladimír Petrík Josef Sivic (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10307.txt.
Introduction
Georgy Ponimatkin and co-authors teach robots by mining how to videos online. The goal is the full three dimensional path of a hand held object through a video. Wild footage is messy, camera work is loose, and the exact object model is unknown. Their pipeline fetches a similar computer model from a huge database, fits it to the frames, and scales it against the scene.
Tracking then smooths the object's path across time. A final optimization bends that path into motions a robot arm can copy. Tests on benchmark sets and fresh hand labeled videos beat prior methods clearly. A real seven axis arm reproduced motions learned from the web.
Catalog record
| CREATOR | Georgy Ponimatkin Martin Cífka Tomáš Souček Médéric Fourmy, Yann Labbé Vladimír Petrík Josef Sivic |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-10307
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
6D Object Pose Tracking in Internet Videos for Robotic Manipulation Georgy Ponimatkin Martin Cífka Tomáš Souček Médéric Fourmy, Yann Labbé Vladimír Petrík Josef Sivic Abstract We seek to extract a temporally consistent 6D pose trajectory of a manipulated object from an Internet instructional video. This is a challenging set-up for current 6D pose estimation methods due to uncontrolled capturing conditions, subtle but dynamic object motions, and the fact that the exact mesh of the manipulated object is not known. To address these challenges, we present the following contributions. First, we develop a new method that estimates the 6D pose of any object in the input image without prior knowledge of the object itself. The method proceeds by (i) retrieving a CAD model similar to the depicted object from a large-scale model database, (ii) 6D aligning the retrieved CAD model with the input image, and (iii) grounding the absolute scale of the object with respect to the scene. Second, we extract smooth 6D object trajectories from Internet videos by carefully tracking the detected objects across video frames. The extracted object trajectories are then retargeted via trajectory optimization into the configuration space of a robotic manipulator. Third, we thoroughly evaluate and ablate our 6D pose estimation method on YCB-V and HOPE-Video datasets as well as a new dataset of instructional videos manually annotated with approximate 6D object trajectories. We demonstrate significant improvements over existing state-of-the-art RGB 6D pose estimation methods. Finally, we show that the 6D object motion estimated from Internet videos can be transferred to a 7-axis robotic manipulator both in a virtual simulator as well as in a real world set-up. We also successfully apply our method to egocentric videos taken from the EPIC-KITCHENS dataset, demonstrating potential for Embodied AI applications. (a) input video (b) retrieved mesh (c) 6D pose trajectory (d) robot trajectory Figure 1: Robotic manipulation guided by an instructional video. (a) Given an instructional video from the Internet, our approach: (b) retrieves a visually similar mesh for the manipulated object from a large mesh database, (c) estimates the approximate 6D pose trajectory of the object across video frames, and (d) transfers the object trajectory onto a 7-axis robotic manipulator. 1 Introduction In this work, we make a step towards learning robotic object manipulation skills from Internet instructional videos, as illustrated in Figure 1 . This is a challenging problem as Internet instructional videos present uncontrolled capturing conditions where people manipulate a large variety of objects often in a dynamic fashion. While humans can easily learn even complex manipulation tasks from one or more instructional videos, building machines with similar cognitive and sensory-motor capabilities remains a major open challenge. Yet, such capabilities would enable tapping into millions of instructional videos available online ( Miech et al., 2019 ) . Current methods for learning object manipulation skills from instructional videos have focused on learning manipulation for stick-like objects (e.g., hammer, scythe, or spade) ( Li et al., 2022 ; Zorina et al., 2021 ) , require tens or even hundreds of training examples captured from multiple viewpoints ( Sermanet et al., 2018 ) or focus on recovering a static pose of the manipulated object ( Patel et al., 2022 ; Wu et al., 2024 ) . In contrast, we leverage the recent progress in 6D object pose estimation and object tracking, and develop a novel method for 6D object pose estimation that allows for extracting temporally consistent 6D object trajectories from uncontrolled Internet videos without any prior knowledge about the object itself. Despite being simple and training-free, the method can extract object trajectories that can be transferred to a robotic manipulator, making a significant step towards large-scale learning of robotic manipulation skills from Internet’s instructional videos. In detail, we make the following contributions: • First, we develop a method that estimates the 6D pose of any object in an image without prior knowledge of the object itself. The method works by (i) retrieving a CAD model similar to the object in the image from a large-scale database, (ii) aligning the retrieved CAD model with the object, and (iii) grounding the object scale with respect to the scene. • Second, we extract smooth 6D object trajectories from Internet videos by carefully tracking the detected objects across video frames. The extracted 6D object trajectories are then retargeted via trajectory optimization into the configuration space of a robotic manipulator. • Third, we thoroughly evaluate and ablate our 6D pose estimation method on YCB-V and HOPE-Video datasets as well as a new dataset of instructional videos manually annotated with approximate 6D object trajectories. We demonstrate significant improvements over existing state-of-the-art RGB 6D pose estimation methods. • Finally, we show that the 6D object motion estimated from Internet videos can be transferred to a 7-axis robotic manipulator both in a virtual simulator as well as in a real world robotic set-up. Additionally, we apply our method to egocentric videos taken from the EPIC-KITCHENS dataset, showing our method can be useful in various Embodied AI applications. 2 Related Work Object instance detection and mesh retrieval. Image based pose estimation methods are usually decomposed into successive steps: object instance detection, often paired with mesh identification, followed by pose estimation. When the depicted objects all belong to a relatively small dataset ( Xiang et al., 2018 ; Hodan et al., 2017 ; Tyree et al., 2022 ) , most methods train an off-the-shelf 2D object detector, such as FasterRCNN Ren et al. (2016) ; Li et al. (2020) , MaskRCNN ( He et al., 2017 ; Labbé et al., 2020 ) or YOLOV8 ( Jocher et al., 2023 ; Wang et al., 2021 ) , using synthetic training data ( Denninger et al., 2023 ) . These supervised detectors achieve very high performance on standard benchmarks ( Hodan et al., 2024 ) but are limited by the need of retraining for each new object instance. More recently, open-ended object detectors have appeared thanks to the advent of powerful foundation models ( Ke et al., 2023 ; Oquab et al., 2024 ) . For a given set of textured meshes, CNOS ( Nguyen et al., 2023 ) proposes an onboarding phase where the known meshes are rendered across a set of viewpoints that are encoded with powerful image features such as ( Oquab et al., 2024 ) . At run time, candidate objects are segmented in the input RGB image using an object segmenter, such as SAM ( Kirillov et al., 2023 ; Zhao et al., 2023 ) , and encoded using the same image features. The depicted object instance is then matched to one of the known meshes whose image encoding is the most similar with the instance features across views. Recent advancements include reducing the number of false detections ( Lu et al., 2024 ) by improving the instance segmentation ( Nguyen et al., 2023 ) using ( Liu et al., 2024b ) , or developing better view-level feature aggregation techniques such as Foreground Feature Averaging ( Kotar et al., 2023 ) . Model-based 6D pose estimation. Recent progress in deep-learning based 6D pose estimation has been focused on increasing precision of alignment-based methods ( Wang et al., 2021 ; Labbé et al., 2020 ) .