Failure Forecasting Boosts Robustness of Sim2Real Rhythmic Insertion Policies
Yuhan Liu, Xinyu Zhang, Haonan Chang, Abdeslam Boularias · 2025
Read Failure Forecasting Boosts Robustness of Sim2Real Rhythmic Insertion Policies by Yuhan Liu, Xinyu Zhang, Haonan Chang, Abdeslam Boularias (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06519.txt.
Introduction
Yuhan Liu and three co-authors make robotic insertion tasks more robust by forecasting failure in this 2025 paper. The task class is rhythmic: a robot must repeat precise insertions, like driving a nut onto a bolt with a wrench, at millimeter accuracy over many cycles. Nut rotation and friction keep shifting the conditions. Their system trains an insertion policy in simulation, then moves it to the real robot. A key choice is to describe the wrench's pose in the nut's own frame, not the robot's, which eases the transfer.
A separate network watches for coming failures. When it predicts a miss, a simple recovery fires: lift the wrench and try again. Real time pose tracking guides each attempt. Tests in simulation and on hardware showed high one time success and, more importantly, steady performance across long repetitive runs. Forecasting failure turns a brittle skill into a durable one.
Catalog record
| CREATOR | Yuhan Liu, Xinyu Zhang, Haonan Chang, Abdeslam Boularias |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-06519
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Failure Forecasting Boosts Robustness of Sim2Real Rhythmic Insertion Policies Yuhan Liu, Xinyu Zhang, Haonan Chang, Abdeslam Boularias Abstract This paper addresses the challenges of Rhythmic Insertion Tasks (RIT), where a robot must repeatedly perform high-precision insertions, such as screwing a nut into a bolt with a wrench. The inherent difficulty of RIT lies in achieving millimeter-level accuracy and maintaining consistent performance over multiple repetitions, particularly when factors like nut rotation and friction introduce additional complexity. We propose a sim-to-real framework that integrates a reinforcement learning-based insertion policy with a failure forecasting module. By representing the wrench’s pose in the nut’s coordinate frame rather than the robot’s frame, our approach significantly enhances sim-to-real transferability. The insertion policy, trained in simulation, leverages real-time 6D pose tracking to execute precise alignment, insertion, and rotation maneuvers. Simultaneously, a neural network predicts potential execution failures, triggering a simple recovery mechanism that lifts the wrench and retries the insertion. Extensive experiments in both simulated and real-world environments demonstrate that our method not only achieves a high one-time success rate but also robustly maintains performance over long-horizon repetitive tasks. For more information please refer to the website: jaysparrow.github.io/rit . I Introduction A Rhythmic Insertion Task ( RIT ) is a type of manipulation task that requires performing precise insertion actions multiple times in succession. It has a wide range of applications, from industrial assembly to everyday tasks. In RIT , a robot must repeatedly insert an object into a receptacle within a single task. A typical example is using a wrench to screw a nut into a bolt, a common industrial assembly task. As shown in Fig. 1 , the robot must repeatedly execute three key steps: aligning the wrench with the nut, pushing down (inserting), and rotating. In this paper, we focus on solving RIT by studying the task of screwing a nut into a bolt using a wrench to better understand the challenges and solutions for general repeated insertion tasks. There are two major challenges in RIT . The first challenge is achieving high-precision insertion. Insertion tasks require millimeter accuracy. Unlike traditional insertion tasks, where the receptacle is fixed, in tasks such as screwing a nut into a bolt, the nut is not fixed but can rotate along the bolt due to friction. This added degree of freedom significantly increases the difficulty of insertion. The second challenge is ensuring consistent success over multiple repetitions. A single successful insertion is not enough—long-horizon performance requires a robust failure recovery mechanism. Without proper failure recovery, even if the one-time success rate exceeds 90%, the overall success rate collapses rapidly as the number of repetitions increases. To address RIT effectively, a robust single-insertion policy and a reliable recovery strategy are essential. Research on one-time insertion tasks has identified three main approaches: (1) visual-tracking alignment with predefined trajectory searching [ 1 , 2 ] , (2) visual-tracking alignment combined with learning-based searching [ 3 , 4 , 5 ] , and (3) end-to-end visual policies [ 6 , 7 , 8 ] . While the tracking-based predefined trajectory method is transferable, it is limited by fixed search patterns, and end-to-end visual policies require extensive fine-tuning on real-world data. A hybrid approach using learning-based strategies with a 6D pose tracker balances adaptability and sim-to-real transferability. Based on this insight, we build on top of IndustReal [ 3 ] for the challenging task of screwing a nut in a bolt with a wrench. We train a pose-based insertion policy in simulation, where nut and wrench poses are directly available, and deploy it in the real world using the 6D pose tracker FoundationPose [ 9 ] . Moreover, we find that representing the wrench’s pose in the nut’s coordinate frame—rather than the robot’s frame—substantially improves sim-to-real performance by a large margin. Fig. 1: Robot executing wrench-based nut insertion: employing a trained neural network to forecast potential failures, the system autonomously corrects its trajectory and retries until the nut is fully inserted, exemplifying enhanced robustness of rhythmic insertion policies. However, achieving a high one-time success rate is not sufficient for RIT . Due to the repetitive nature of the task, a failure forecasting and recovery strategy is crucial. Failure recovery mechanisms have been widely studied in long-horizon manipulation [ 10 ] . Failure recovery consists of two key components: the recovery policy and the recovery triggering-events. Some approaches explicitly collect failure data to train a recovery policy [ 11 , 12 ] . However, for RIT , a simple recovery strategy—lifting and retrying the insertion—is often sufficient. In this paper, we adopt this straightforward recovery policy. Regarding failure detection, there are two common approaches: anomaly detection [ 13 , 14 ] and failure prediction [ 15 , 16 ] . Anomaly detection models normal system behavior and identifies deviations, making it suitable when ample success data is available. In contrast, failure prediction, which directly predicts if a trajectory is going to end in a failure, is more appropriate for our case since we have both success and failure data from simulation training. In practice, we train a network to detect the potential failure within the current execution, and perform the lift-up recovery once a failure is predicted. In summary, we propose a sim-to-real policy in object coordinates with failure recovery. The proposed method has two components: (1) a reinforcement learning policy for single-insertion tasks that takes the wrench’s pose in the nut’s coordinate frame as input and outputs the 6D displacement of the wrench, and (2) a failure recovery module that lifts the wrench to retry the insertion when needed. In real-world experiments, we use the 6D pose tracker FoundationPose [ 9 ] to track object and wrench poses and a failure forecasting method based on simulation data to predict potential failures. Extensive experiments validate the effectiveness of our method in both simulation and real environments. Our contributions can be summarized as follows: 1. To the best of our knowledge, the proposed method is the first robust policy specifically designed to address the repeated insertion task. 2. We improve the sim-to-real transferability by modifying IndustReal ’s coordinate representation from the robot frame to the receptacle’s frame. Extensive experiments in both simulation and real-world environments validate the effectiveness of this modification 3. We systematically evaluate multiple failure forecasting methods and their impact on RIT performance through extensive experiments in both simulation and real-world settings. II Related Works Tight Insertion . Object insertion under (sub-)millimeter tolerances has been extensively studied in the literature. Data-driven approaches learn end-to-end insertion policies from high-dimensional visual inputs [ 6 , 7 , 8 ] . These methods usually require extra online fine-tuning [ 6 ] or other sensor modalities [ 7 , 8 ] to deploy in the real-world. Tracking-based methods often demonstrate stronger transfer ability in real-world deployment. Combinations of various searching patterns and force-feedback controllers for insertion have also been studied [ 1 , 2 ] . While fixed patterns are resilient to calibration errors and external disturbances on stationary receptacles, they are not robust over moving objects (e.g., rotating nuts) as in our task. Recently, RL has became a popular choice for insertion policy learning [ 3 , 4 , 5 ] . Among these works, IndustReal