Real-world Reinforcement Learning from Suboptimal Interventions

Yinuo Zhao, Huiqian Jin, Lechun Jiang, Xinyi Zhang, Kun Wu, Pei Ren, Zhiyuan Xu, Zhengping Che, Lei Sun, Dapeng Wu, Chi Harold Liu, Jian Tang · 2025

Read Real-world Reinforcement Learning from Suboptimal Interventions by Yinuo Zhao, Huiqian Jin, Lechun Jiang, Xinyi Zhang, Kun Wu, Pei Ren, Zhiyuan Xu, Zhengping Che, Lei Sun, Dapeng Wu, Chi Harold Liu, Jian Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-24288.txt.

Introduction

Yinuo Zhao and five colleagues teach robots with imperfect human help. Real-world reinforcement learning lets robots learn manipulation from their own trials. Human interventions during training are usually treated as perfect. Even experts cannot act optimally everywhere, though. They make mistakes in some states. Blindly imitating them caps what the robot can learn.

Their algorithm, SiLRI, weighs help state by state. The problem is cast as constrained optimization. At each state, the constraint depends on how uncertain the human help is there. A state-wise Lagrange multiplier enforces the balance. Policy and multiplier are optimized jointly toward a saddle point. The system runs on a human-as-copilot teleoperation setup. Diverse real manipulation tasks tested it. Time to reach a high success rate fell by at least half against a leading rival. Long-horizon tasks reached full success where other methods struggled.

Catalog record

CREATORYinuo Zhao, Huiqian Jin, Lechun Jiang, Xinyi Zhang, Kun Wu, Pei Ren, Zhiyuan Xu, Zhengping Che, Lei Sun, Dapeng Wu, Chi Harold Liu, Jian Tang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-24288

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Real-world Reinforcement Learning from Suboptimal Interventions Yinuo Zhao, Huiqian Jin, Lechun Jiang, Xinyi Zhang, Kun Wu, Pei Ren, Zhiyuan Xu, Zhengping Che, Lei Sun, Dapeng Wu, Chi Harold Liu, Jian Tang Abstract Real-world reinforcement learning (RL) offers a promising approach to training precise and dexterous robotic manipulation policies in an online manner, enabling robots to learn from their own experience while gradually reducing human labor. However, prior real-world RL methods often assume that human interventions are optimal across the entire state space, overlooking the fact that even expert operators cannot consistently provide optimal actions in all states or completely avoid mistakes. Indiscriminately mixing intervention data with robot-collected data inherits the sample inefficiency of RL, while purely imitating intervention data can ultimately degrade the final performance achievable by RL. The question of how to leverage potentially suboptimal and noisy human interventions to accelerate learning without being constrained by them thus remains open. To address this challenge, we propose SiLRI , a state-wise Lagrangian reinforcement learning algorithm for real-world robot manipulation tasks. Specifically, we formulate the online manipulation problem as a constrained RL optimization, where the constraint bound at each state is determined by the uncertainty of human interventions. We then introduce a state-wise Lagrange multiplier and solve the problem via a min-max optimization, jointly optimizing the policy and the Lagrange multiplier to reach a saddle point. Built upon a human-as-copilot teleoperation system, our algorithm is evaluated through real-world experiments on diverse manipulation tasks. Experimental results show that SiLRI effectively exploits human suboptimal interventions, reducing the time required to reach a 90% success rate by at least 50% compared with the state-of-the-art RL method HIL-SERL, and achieving a 100% success rate on long-horizon manipulation tasks where other RL methods struggle to succeed. Project website: https://silri-rl.github.io/ † † footnotetext: 🖂 Corresponding author: Jian Tang ([email protected]). \dagger Project leaders: Zhengping Che and Zhiyuan Xu. This work is done during Yinuo Zhao’s internship at Beijing Innovation Center of Humanoid Robotics. I Introduction Vision-based robotic manipulation has recently achieved remarkable progress in real-world applications. Either Transformer-based methods (e.g., ACT [ 28 ] ) or diffusion-based approaches (e.g., DP [ 5 ] ) achieves strong performance by leveraging expert demonstrations collected through teleoperation [ 27 , 24 , 25 ] . Despite their success, Imitation Learning (IL) paradigms remain constrained by their heavy dependence on large amounts of high-quality demonstrations, which require substantial human effort. In contrast, reinforcement learning (RL) offers a more autonomous alternative by enabling robots to explore and optimize policies in their own way, thereby reducing dependence on human supervision. However, unlike simulation-based RL, where agents can explore freely and at low cost, practical deployment in complex real-world scenarios is still limited by RL’s poor sample efficiency and safety concerns. (a) Fig. 1 : Entropy of Human Interventions Across States. Entropy is estimated with a multivariate normal distribution model. In low-entropy states (blue region), human operators intervene consistently and confidently, whereas in high-entropy states (green region), their interventions are inconsistent, indicating uncertainty. (a) Fig. 2 : SiLRI Enables Effective Real-world RL from Suboptimal Interventions. Left: A human-as-copilot teleoperation system that enables seamless human intervention. Right: The overall optimization objective of SiLRI. Relaxing the constraint in high-entropy states enables the learned policy \pi to converge to a policy that outperforms human behavior policy \beta . Human-in-the-loop Reinforcement Learning (HIL-RL) [ 17 , 3 ] has recently emerged as a promising paradigm for tackling complex real-world robot manipulation tasks, achieving near human-level performance with highly dexterous skills. HIL-RL typically leverages three types of data: online policy transitions, online human interventions, and offline human demonstrations. For clarity, we group the first two as online data and the latter two as intervention data. In practice, due to the high-dimensional exploration space and sparse reward signals in real-world environments, existing HIL-RL methods typically store online and intervention data in separate buffers (with online human interventions duplicated) and sample from them uniformly during RL updates. This strategy increases the sampling probability of scarce yet informative successful transitions from intervention data, which is a key factor underlying their current success. However, intervention data are often integrated into the RL pipeline without careful consideration of their underlying properties. We illustrate this issue with the following example. As shown in Fig. 1 , we plot the policy entropy along a human-collected trajectory in the Hang Chinese Knot task. The entropy is estimated using a multivariate normal distribution fitted to 20 trajectories of human demonstrations. We observe that in low-entropy states, such as when picking up the Chinese knot, humans tend to provide consistent actions (i.e., similar actions under the same state). In contrast, in states that require high precision, humans fail to produce consistent actions, leading to high entropy. This suboptimality reveals a fundamental challenge: different states require different balances between the RL and IL objectives. In low-entropy states, imitation is effective, but in high-entropy states, suboptimal interventions can hinder performance, making the RL objective a more reliable guide. To address this challenge, we propose SiLRI , a S tate-w i se L agrangian R einforcement learning framework from suboptimal I nterventions for real-world robot manipulation training. Specifically, we first formulate a constrained RL optimization problem, where the constraint bound at each state is determined by the corresponding human uncertainty. As shown in the right part of Fig. 2 , in states where humans provide consistent data (low entropy), the learned policy is constrained to stay close to the human behavior policy through increasing Lagrange multiplier \lambda , to improve training efficiency. In high-entropy states, the constraint is relaxed, encouraging the policy to be optimized primarily through its own estimated critic, i.e., RL objective. To realize this behavior, we introduce a state-wise Lagrange multiplier in the imitation term and formulate the problem as a min-max optimization. The policy objective is minimized while the dual objective over the Lagrange multiplier is maximized, enabling the state-wise multiplier to be adaptively adjusted to balance the RL and IL objectives. This design improves RL sample efficiency while ensuring convergence to a policy that can surpass suboptimal intervention data. Building SiLRI on a human-as-copilot teleoperation system, as shown in the left part of Fig. 2 , we conduct extensive real-world experiments on two robotic embodiments. Comparisons against a classical online IL method and state-of-the-art (SOTA) real-world RL methods demonstrate the effectiveness of SiLRI. We summarize our contributions as follows: • We formulate a constrained RL problem that explicitly accounts for human uncertainty across states, enabling more effective use of suboptimal human interventions. • To solve this constrained RL problem, we introduce learnable state-wise Lagrange multipliers and cast the optimization as a min-max problem, allowing the agent to adaptively trade off between RL and IL objectives. • Built on a human-robot collaboration system, our real-world experiments demonstrate that SiLRI reaches a 90% success rate at least 50% faster (corresponding to an 18-minute reduction) than SOTA method HIL-SERL, while achieving a 100% success rate on long-horizon manipulation tasks where other RL methods struggle. II Related Works Imitation Learning for Real-world Manipulations Imitation learning has dominated the end-to-end learning paradigm [ 7 , 1 ] for real-world manipulation for decades. With high-quality demonstrations collected via teleoperation systems (e.g., VR headsets [

Read the complete text