RoboHitch: Learning Visual Affordance from Disordered Keypoints for Hitch Knots Tying

Jiahui Zuo, Boyang Zhang, Fumin Zhang, Fellow, IEEE · 2026

Read RoboHitch: Learning Visual Affordance from Disordered Keypoints for Hitch Knots Tying by Jiahui Zuo, Boyang Zhang, Fumin Zhang, Fellow, IEEE (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24394.txt.

Introduction

Zuo, Boyang Zhang, and Fumin Zhang present RoboHitch, tying hitch knots from human demos with disordered 3D keypoints and RGB images. Knot tying with bendy linear objects is hard: existing methods track ordered keypoints, but repeated bending and crossings cause drift and mismatch. RoboHitch drops the need for explicit order. A Graph Autoencoder pulls geometric features from untracked keypoints, a Convolutional Autoencoder adds visual context, and cross-attention fuses both to predict pick-and-place affordances under occlusion. Real-world tests complete hitch knots even with self-occlusions.

Catalog record

CREATORJiahui Zuo, Boyang Zhang, Fumin Zhang, Fellow, IEEE
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33989
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.24394

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RoboHitch: Learning Visual Affordance from Disordered Keypoints for Hitch Knots Tying Jiahui Zuo, Boyang Zhang, Fumin Zhang, Fellow, IEEE Abstract Robotic manipulation of deformable linear objects (DLOs) presents significant challenges due to complex dynamics and frequent self-occlusions. Existing robotic knot tying methods typically rely on precise topological state tracking with ordered keypoints and explicit edge connectivity. This reliance makes them prone to failures due to tracking drift and topology mismatch caused by repeated bending and crossings during knot formation. To address these limitations, we introduce RoboHitch, a novel framework that learns to perform hitch knot tying from human demonstrations using only disordered 3D keypoints and RGB images. This eliminates the need for explicit topological order, allowing for more flexible manipulation. Our method employs a dynamic Graph Autoencoder to extract geometric features from untracked keypoints, complemented by a Convolutional Autoencoder that captures essential visual context. A bidirectional cross-attention mechanism then fuses these modalities to jointly predict pick and place affordances, facilitating implicit reasoning about the rope’s state and enabling knot tying under occlusion. Real-world experiments demonstrate the effectiveness and generalizability of our approach, successfully completing hitch knots in scenarios with self-occlusions. I Introduction Knotting manipulation of DLOs is widely employed in diverse applications, including surgery, industrial automation, and maritime operations. Knot tying can be classified into three distinct categories: (1) knots, formed within a single rope like overhand knots, (2) bends, connecting between two or more ropes, and (3) hitches, tying a rope to another object. In real-world scenarios, knot-tying is typically performed to fasten a rope to a target object rather than creating an isolated knot. Consequently, this work focuses on robotic hitch knot tying, a task that humans perform with ease, but remains difficult for robotics. A fundamental challenge in robotic knot tying is DLO state representation. Unlike rigid objects, there is no direct representation of the DLO state [ 1 ] . Although RGB images and point clouds are easy to obtain, they are relatively unstructured and often data hungry. To facilitate subsequent motion planning, DLOs are typically represented as ordered lists of keypoints, where the order encodes topology along the rope. This structured prior is useful, but accurate keypoint estimation is difficult during knot formation because self-occlusion can break correspondence and ordering. While previous work has made progress in tracking DLO topology using point cloud registration for overhand knots or simpler configurations [ 11 , 19 , 23 ] , these methods often assume uncrossed DLO initialization and struggle to maintain accurate keypoint tracking from complex self-occlusion states for knot tying [ 3 , 7 ] . Topology mismatch from tracking failure can easily lead to task failure caused by failed grasps. With a DLO state estimate, both model-based [ 19 , 6 , 20 ] and learning-based methods [ 18 , 12 , 14 ] seek the optimal action sequences to complete the task. Model-based pipelines typically require explicit and consistent keypoint ordering (often via markers) to classify knot states [ 9 ] and execute hand-designed primitives to transition states. The reliance on explicit topological states making them sensitive to occlusion-induced tracking errors without markers. Fig. 1: Overview of our study. (a) Our approach learns to tie hitch knots through human demonstrations. (b) Given an RGB image and 3D keypoints of the rope, the model employs bidirectional cross-attention to fuse multimodal features and predict pick, rotate, and place affordances. (c) The robot executes sequential actions until the task is completed. Furthermore, many learning-based methods use RGB or point cloud inputs from easier state estimation and adopt end-to-end frameworks that tightly couple perception and action. This coupling often leads to large quantities of real-world data requirements and makes policies difficult to interpret. As shown in Fig. 1 , we propose a framework that learns hitch knot tying policies from visual observations, without relying on perfectly tracked or ordered topological states. Our key insight is that by fusing disordered 3D rope keypoints and RGB scene image, the model can implicitly reason about the rope’s physical state and infer effective manipulation actions with limited data, even in the presence of occlusions or crossings. This multimodal bias combines geometric and visual cues, thus constraining the complexity of learning and facilitating more efficient model training. Specifically, we employ a Graph Autoencoder (GAE) to encode the topological features from disordered keypoints, while a Convolutional Autoencoder (CAE) processes scene features derived from RGB input. A bidirectional cross-attention mechanism further integrates both modalities to jointly predict pick and place affordances tailored for single-arm robotic knot tying. We collect human demonstration videos to train our policy and evaluate its performance through a series of real-world hitch knot-tying experiments. The results show that our method achieves robust knotting performance across various scenarios, effectively handling challenges related to disordered rope point perception and self-occlusions. Key contributions of this work include the following: • We present a rope state representation that extracts disordered 3D keypoints from a potentially occluded rope, suitable for downstream manipulation without fragile tracking of precise topological order. • We introduce a knotting framework that leverages bidirectional cross-attention to fuse rope geometric features and visual scene features, facilitating the prediction of pick and place affordances for hitch knot tying in visually complex environments. • The framework is experimentally validated through real-world robotic hitch knot tying with a single arm, demonstrating effective performance under self-occlusion. II Related Work II-A State Detection and Tracking of DLOs Accurate DLO topology estimation for knot tying remains challenging, since vision systems often fail to maintain ordered correspondence under self-occlusion and complex self-crossings. Some works simplify the problem by assuming the rope is fully visible to track during manipulation [ 25 , 27 , 26 ] . Significant progress has been made in developing visual tracking methods that can handle some occlusions and self-crossings [ 4 , 22 , 23 ] . A common pipeline first extracts an uncrossed initial state via segmentation and refinement [ 3 , 7 ] , subsequently tracks the consistent rope points across frames using non-rigid registration techniques such as Coherent Point Drift (CPD) [ 11 , 19 , 23 ] or predictive models [ 5 ] . However, during knot formation, persistent bending and repeated crossings can cause drift and topology mismatch, leading to incorrect state estimates and downstream manipulation failures [ 23 , 5 ] . Unlike these tracking approaches, our method foregoes the need for a perfectly ordered topological sequence. Instead, we demonstrate that a robust knot-tying policy can be learned directly from a disordered set of keypoints inputs, which is more reliable in self-occluded scenarios. II-B Robotic Knotting of Ropes Robotic knot-tying research can be broadly categorized into two main approaches, model-based methods that utilize estimated geometric information [ 20 , 6 , 10 ] and learning-based approaches that employ reinforcement or imitation learning [ 18 , 12 , 14 ] . Model-based methods typically rely on physically accurate simulations or geometric reasoning based on the estimated state of the rope. Some approaches plan actions by computing the similarity between the current state and a predefined goal state using registration techniques like CPD [ 20 ] . Recent study has also explored the application of Reidemeister Moves principles from knot theory [

Read the complete text