Confidence-Guided Human-AI Collaboration: Reinforcement Learning with Distributional Proxy Value Propagation for Autonomous Driving

Zeqiao Li, Yijing Wang, Haoyu Wang, Zheng Li, Peng Li, Zhiqiang Zuo, Chuan Hu · 2025

Read Confidence-Guided Human-AI Collaboration: Reinforcement Learning with Distributional Proxy Value Propagation for Autonomous Driving by Zeqiao Li, Yijing Wang, Haoyu Wang, Zheng Li, Peng Li, Zhiqiang Zuo, Chuan Hu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03568.txt.

Introduction

Zeqiao Li and co-authors present a 2025 method for self-driving cars that learn with a human guide on call. Learned driving faces unsafe trials and drift from the data it trained on, while constant human help is dear and slow. Their frame calls the human in only when its own confidence runs low.

A spread-based value method learns the guide's intent from few inputs, which steadies the early learning. A shared control step then blends the guided policy with a self-taught one. A confidence check flips between them, so the car can pass its teacher while staying safe.

Catalog record

CREATORZeqiao Li, Yijing Wang, Haoyu Wang, Zheng Li, Peng Li, Zhiqiang Zuo, Chuan Hu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03568

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Confidence-Guided Human-AI Collaboration: Reinforcement Learning with Distributional Proxy Value Propagation for Autonomous Driving Zeqiao Li, Yijing Wang, Haoyu Wang, Zheng Li, Peng Li, Zhiqiang Zuo, Chuan Hu Abstract Autonomous driving promises significant advancements in mobility, road safety and traffic efficiency, yet reinforcement learning and imitation learning face safe-exploration and distribution-shift challenges. Although human-AI collaboration alleviates these issues, it often relies heavily on extensive human intervention, which increases costs and reduces efficiency. This paper develops a confidence-guided human-AI collaboration (C-HAC) strategy to overcome these limitations. First, C-HAC employs a distributional proxy value propagation method within the distributional soft actor-critic (DSAC) framework. By leveraging return distributions to represent human intentions C-HAC achieves rapid and stable learning of human-guided policies with minimal human interaction. Subsequently, a shared control mechanism is activated to integrate the learned human-guided policy with a self-learning policy that maximizes cumulative rewards. This enables the agent to explore independently and continuously enhance its performance beyond human guidance. Finally, a policy confidence evaluation algorithm capitalizes on DSAC’s return distribution networks to facilitate dynamic switching between human-guided and self-learning policies via a confidence-based intervention function. This ensures the agent can pursue optimal policies while maintaining safety and performance guarantees. Extensive experiments across diverse driving scenarios reveal that C-HAC significantly outperforms conventional methods in terms of safety, efficiency, and overall performance, achieving state-of-the-art results. The effectiveness of the proposed method is further validated through real-world road tests in complex traffic conditions. The videos and code are available at: https://github.com/lzqw/C-HAC. I Introduction Autonomous driving technology is revolutionizing transportation by offering enhanced mobility, improved safety, increased traffic efficiency, and reduced environmental impacts. These advancements have garnered significant attention from both academia and industry [ 1 , 2 , 3 , 4 ] . Recent advances in artificial intelligence (AI) have propelled learning-based policies to the forefront of autonomous driving research, providing an alternative to traditional modular approaches that segment the driving system into distinct components such as perception, localization, planning, and control [ 5 , 6 ] . Learning-based policies, particularly end-to-end approaches, have demonstrated potential in enhancing the perception and decision-making capabilities of AI systems in autonomous vehicles (AVs) [ 7 ] . To further enhance learning-based approaches in autonomous driving, reinforcement learning (RL), imitation learning (IL) and human-AI collaboration (HAC) learning have been extensively explored. RL is a powerful paradigm for training autonomous systems through trial-and-error interactions with the environment. It enables agents to establish causal relationships among observations, actions, and outcomes [ 8 ] . A reward function guides the learning process and encodes desired behaviors, allowing extensive exploration of the environment to maximize cumulative rewards. This approach often yields robust policies capable of handling complex tasks [ 9 , 10 , 11 ] . Nevertheless, several challenges restrict the applicability of RL in safety-critical scenarios. Designing a reward function that aligns with diverse human preferences is nontrivial, and flawed reward functions may lead to biased, misguided, or undesirable behaviors [ 12 , 13 , 14 ] . Moreover, the exploratory nature of RL frequently induces unsafe situations during both training and testing. The low sample efficiency in agent-environment interactions further exacerbates computational costs and prolongs training [ 15 , 16 ] . In addition, although policies trained exclusively with RL can be technically safe, they often lack the natural, human-like behaviors critical for coordinating with other vehicles in real-world driving scenarios [ 17 ] . Therefore, there is a pressing need for approaches that mitigate these limitations—particularly unsafe exploration, reward design pitfalls, and the lack of human-like behavior—to fully realize RL’s potential in autonomous driving. IL is a promising approach for learning driving policies by replicating human driving behavior through demonstrated actions. Prominent methods include behavior cloning (BC) [ 18 ] and inverse reinforcement learning (IRL) [ 19 ] . Leveraging expert demonstrations allows novice agents to avoid hazardous interactions during training, enhancing safety. Techniques such as BC and offline RL train agents on pre-existing datasets without environment interaction, further reducing risk [ 20 , 21 , 22 , 23 , 24 , 25 ] , while IRL infers a reward function from demonstrations to guide desirable behaviors [ 26 ] . Despite these strengths, IL faces significant challenges. Distributional shift can cause compounding errors, pushing the agent away from its training distribution and resulting in control failures [ 27 , 28 ] . Moreover, IL’s success heavily depends on the quality of demonstrations, leaving it vulnerable to task variations [ 29 ] . Rare or atypical scenarios, often underrepresented in training data, can lead to erratic policy responses, especially in complex environments where expert data may be scarce or suboptimal [ 30 ] . Consequently, IL-based methods demand solutions that mitigate distributional shift, cope with suboptimal or limited demonstrations, and adapt to diverse driving conditions. Human-AI collaboration (HAC) leverages human expertise to enhance safety and efficiency. Methods such as DAgger [ 31 ] and its extensions [ 32 , 33 , 34 ] periodically request expert demonstrations to correct compounding errors in IL, while expert intervention learning (EIL) [ 35 ] and intervention weighted regression (IWR) [ 36 ] rely on human operators to intervene during exploration for safer state transitions. Other approaches integrate human evaluative feedback [ 37 , 38 , 39 ] or partial demonstrations with limited interventions, as in HACO [ 40 ] , to guide the learning process while reducing human effort. Proxy value propagation (PVP) [ 41 ] uses active human input to learn a proxy value function encoding human intentions, demonstrating robust performance across diverse tasks and action spaces. Despite these advances, many HAC methods assume near-optimal human guidance, which can be prohibitively expensive or suboptimal in practice [ 42 ] . Humans may adopt conservative strategies in ambiguous scenarios, such as decelerating behind a slower vehicle for safety, even when a more effective approach (e.g., overtaking) might exist. Furthermore, continuous oversight places a heavy burden on human operators, hindering large-scale deployment [ 19 ] , while insufficient utilization of autonomous exploration data constrains overall learning efficiency [ 43 ] . Balancing human involvement, agent safety, and learning efficiency thus remains a central challenge in HAC research. To reduce reliance on human guidance while ensuring efficient and safe policy learning, we propose a confidence-guided human-AI collaboration (C-HAC) strategy in this paper. It operates in two stages: a human-guided learning phase and a subsequent RL enhancement phase. In the former, Distributional proxy value propagation (D-PVP) encodes human intentions in the Distributional soft actor-critic (DSAC) algorithm. For the latter, the agent refines its policy independently. A shared control mechanism combines the learned human-guided policy with a self-learning policy, enabling exploration beyond human demonstrations. Additionally, a policy confidence evaluation algorithm leverages DSAC’s return distributions to switch between human-guided and self-learning policies, ensuring safety and reliable performance. Extensive experiments show that C-HAC quickly acquires robust driving skills with minimal human input and continues improving without ongoing human intervention, outperforming conventional RL, IL, and HAC methods in safety, efficiency, and overall results. The main contributions of this paper are as follows: • Distributional proxy value propagation (D-PVP): A novel method integrating PVP into the DSAC framework is suggested. D-PVP enables the agent to learn effective driving policies with relatively

Read the complete text