RoboTAG: End-to-end Robot Configuration Estimation via Topological Alignment Graph
Yifan Liu, Fangneng Zhan, Wanhua Li, Haowen Sun, Katerina Fragkiadaki, Hanspeter Pfister · 2025
Read RoboTAG: End-to-end Robot Configuration Estimation via Topological Alignment Graph by Yifan Liu, Fangneng Zhan, Wanhua Li, Haowen Sun, Katerina Fragkiadaki, Hanspeter Pfister (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07717.txt.
Introduction
Yifan Liu, Fangneng Zhan, and Wanhua Li estimate a robot full configuration from one ordinary photo. Knowing joint pose from a single camera view helps robots work beside people, but labeled training images are scarce and two-dimensional shortcuts ignore three-dimensional structure. Simulators help, yet the gap to real scenes persists.
RoboTAG aligns a topological graph of the robot across two branches, one working in image space and one carrying three-dimensional priors. The branches evolve together, reducing dependence on labels. Graph nodes track robot parts and their connections explicitly. Better pose from cheap cameras could widen where robots safely operate outside staged labs.
Catalog record
| CREATOR | Yifan Liu, Fangneng Zhan, Wanhua Li, Haowen Sun, Katerina Fragkiadaki, Hanspeter Pfister |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-07717
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RoboTAG: End-to-end Robot Configuration Estimation via Topological Alignment Graph Yifan Liu, Fangneng Zhan, Wanhua Li, Haowen Sun, Katerina Fragkiadaki, Hanspeter Pfister Abstract Estimating robot pose from a monocular RGB image is a challenge in robotics and computer vision. Existing methods typically build networks on top of 2D visual backbones and depend heavily on labeled data for training, which is often scarce in real-world scenarios, causing a sim-to-real gap. Moreover, these approaches reduce the 3D-based problem to 2D domain, neglecting the 3D priors. To address these, we propose Robot Topological Alignment Graph (RoboTAG), which incorporates a 3D branch to inject 3D priors while enabling co-evolution of the 2D and 3D representations, alleviating the reliance on labels. Specifically, the RoboTAG consists of a 3D branch and a 2D branch, where nodes represent the states of the camera and robot system, and edges capture the dependencies between these variables or denote alignments between them. Closed loops are then defined in the graph, on which a consistency supervision across branches can be applied. This design allows us to utilize in-the-wild images as training data without annotations. Experimental results demonstrate that our method is effective across robot types, highlighting its potential to alleviate the data bottleneck in robotics. 1 Introduction Robot pose estimation from monocular RGB images is a fundamental task in robotics and computer vision. Establishing a direct mapping between visual signals and robot configurations enables critical downstream applications such as human-robot interaction Christen et al. 2023 ; Svenstrup et al. 2009 ; Xu et al. 2020 and multi-robot collaboration Li et al. 2022 ; Papadimitriou et al. 2022 ; Rizk et al. 2019 , while also opening new possibilities for automatic robot data annotation and the integration of vision foundation models in robotics, including diffusion models Ho et al. 2022 ; Blattmann et al. 2023 ; Yang et al. 2025 for long-term planning Liu et al. 2024b ; Zhen et al. 2025 . However, this task is challenging due to the abstractness of robot configurations Ban et al. 2024 . Existing methods Labbé et al. 2021 ; Ban et al. 2024 ; Goswami et al. 2025 typically build networks on top of 2D visual backbones and depend heavily on labeled data for training, which makes deployment in real-world scenarios difficult. Moreover, these approaches reduce the inherently 3D problem to the 2D domain, neglecting rich priors available in pre-trained 3D models and suffering from the inherent spatial ambiguity of 2D representations Marr 2010 , thus limiting performance. Figure 1 : The intuition of RoboTAG. (a) Existing works predict every state (square) of the camera and robot system independently, and apply supervision on each of them. (b) Our method constructs a topological graph, RoboTAG, by introducing a 3D branch (blue) with forward edges (solid arrows) and alignment edges (dotted arrows) connecting system states. Multiple forward edges to a state indicate it depends on several others. Closed loops are defined in this topology to enable 2D-3D consistency supervision. To tackle these challenges, we propose Robo t T opological A lignment G raph ( RoboTAG ): A 3D prior is injected with a topological structure, which enables the co-evolution Liu et al. 2024a of 2D and 3D representations, thereby alleviating reliance on labeled data. The name of RoboTAG reflects two aspects of our approach beyond abbreviation: (1) the framework acts as semantic tags for unannotated data through 2D-3D alignment, and (2) the end-to-end model can generate labels for in-the-wild robot videos. Specifically, RoboTAG consists of dual branches: a 3D branch and a 2D branch, which constructs a graph in a unified system of a robot and cameras. Nodes in the graph represent system state variables (camera extrinsics, robot joint angles, keypoints, etc.), while edges connecting these nodes are defined in two types: (a) forward edges that capture dependencies between variables, and (b) alignment edges that denote correspondences between equivalent nodes across branches. As shown in Figure 1 , these edges form a topology containing closed loops defined with specific constraints, enabling 2D-3D consistency supervision over the alignment edges. These closed loops strengthen the graph topology and enable alignment across branches. With RoboTAG, the 2D and 3D branches are deeply intertwined, enabling co-evolution of both backbones. This design allows us to effectively utilize in-the-wild images as training data without annotations. Compared to existing methods, our approach not only improves performance by leveraging 3D priors but also alleviates the need for labeled ground truth when deploying in real-world scenarios. Extensive experimental results demonstrate the effectiveness of our method across various robot types and in both real-world and synthetic settings. This work opens new possibilities for alleviating the data bottleneck in robotics through self-supervised learning approaches. Our contributions are summarized as follows: • We introduce Robot Topological Alignment Graph (RoboTAG) for robot pose estimation, which effectively incorporates 3D priors and enables 2D-3D alignment. • We propose a novel graph topology with closed loops that enables 2D-3D consistency supervision, allowing co-evolution of 2D and 3D representations. • We achieve state-of-the-art performance across robot types and demonstrate the potential of self-supervised learning to mitigate the data bottleneck in robotics. 2 Related Work Robot Hand-Eye Calibration. Traditional methods for robot hand-eye calibration Garrido-Jurado et al. 2014 ; Olson 2011 ; Fiala 2005 typically rely on a set of known correspondence points between the robot’s end-effector and the camera. With known joint angles, the 3D positions of the correspondence points can be computed, and an optimization problem Park and Martin 1994 ; Ilonen and Kyrki 2011 ; Yang and Illingworth 1994 is solved to find the camera-to-robot transformation. This, however, requires a set of known correspondence points in different robot configurations, which is hard to obtain in real-world scenarios. Recent learning-based approach represented by Lee et al. 2020 tackle this problem by predicting the keypoints using networks and then do camera pose estimation with Perspective-n-Points (PnP) algorithms Li et al. 2012 ; Du et al. 2021 ; Chen et al. 2020 . Further works like Han et al. 2024 ; Zhong et al. 2023 ; Lu et al. 2023 ; Tian et al. 2023 improve performance by introducing modern architectures or modules like feature fusion or temporal attention. Robot Pose Estimation. Robot hand-eye calibration problems are often solved in the assumption of known robot joint angles, which limits the applicability of these methods in real-world settings like multi-robot collaboration. To address this, RoboPose Labbé et al. 2021 proposed a refinement-based method that estimates the robot joint angles and camera pose by iteratively rendering and comparing, which requires significant computation. RoboKeyGen Tian et al. 2024 utilizes a diffusion model to lift predicted 2D keypoints to 3D. RoboPEPP Goswami et al. 2025 further introduces a masked autoencoder(MAE) He et al. 2021 during training to give the network a robust understanding of the robot and camera system. These methods, however, tackle the problem in a 2D domain and do not leverage rich 3D information in this problem. Also, they rely heavily on labeled data for training, which is often not abundant in real-world deployments. Figure 2 : Overview of the proposed method. The framework consists of a 3D branch and a 2D branch, which are deeply intertwined as a topological graph. The nodes represent the states of