Interactive Double Deep Q-network: Integrating Human Interventions and Evaluative Predictions in Reinforcement Learning of Autonomous Driving

Alkis Sygkounas 1 , Ioannis Athanasiadis 2 , Andreas Persson 1 , Michael Felsberg 2 , Amy Loutfi 1 · 2025

Read Interactive Double Deep Q-network: Integrating Human Interventions and Evaluative Predictions in Reinforcement Learning of Autonomous Driving by Alkis Sygkounas 1 , Ioannis Athanasiadis 2 , Andreas Persson 1 , Michael Felsberg 2 , Amy Loutfi 1 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-01440.txt.

Introduction

Alkis Sygkounas and co-authors teach self-driving cars with a human riding along during training, in a 2025 paper. Their method, iDDQN, folds a person's takeovers and hints straight into the learning rule of a driving agent. The goal is a policy shaped jointly by machine trials and human judgment, for a task where errors carry real danger.

A companion test replays each drive as if no human had stepped in, scoring how much the help actually bought. In simulated driving, the method beat cloning of human drives, guided data methods, and plain deep learning at using expert help. The authors pitch it as a safer way to grow driving skill.

Catalog record

CREATORAlkis Sygkounas 1 , Ioannis Athanasiadis 2 , Andreas Persson 1 , Michael Felsberg 2 , Amy Loutfi 1
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-01440

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Interactive Double Deep Q-network: Integrating Human Interventions and Evaluative Predictions in Reinforcement Learning of Autonomous Driving Alkis Sygkounas 1 , Ioannis Athanasiadis 2 , Andreas Persson 1 , Michael Felsberg 2 , Amy Loutfi 1 Abstract Integrating human expertise with machine learning is crucial for applications demanding high accuracy and safety, such as autonomous driving. This study introduces Interactive Double Deep Q-network (iDDQN), a Human-in-the-Loop (HITL) approach that enhances Reinforcement Learning (RL) by merging human insights directly into the RL training process, improving model performance. Our proposed iDDQN method modifies the Q-value update equation to integrate human and agent actions, establishing a collaborative approach for policy development. Additionally, we present an offline evaluative framework that simulates the agent’s trajectory as if no human intervention to assess the effectiveness of human interventions. Empirical results in simulated autonomous driving scenarios demonstrate that iDDQN outperforms established approaches, including Behavioral Cloning (BC), HG-DAgger, Deep Q-Learning from Demonstrations (DQfD), and vanilla DRL in leveraging human expertise for improving performance and adaptability. I INTRODUCTION Achieving autonomous driving remains a key challenge in developing intelligent vehicles capable of reliably perceiving their environment, making real-time decisions, and executing precise control in dynamic and uncertain environments. Deep Reinforcement Learning (DRL) has demonstrated great potential in autonomous driving [ 2 , 18 ] , as well as in high-dimensional control problems [ 8 ] , reward optimization [ 26 ] , and decision-making and control under dynamic conditions [ 14 ] . However, a persistent challenge remains in effectively integrating human expertise into DRL models to enhance safety, adaptability, and interpretability. Human-in-the-Loop (HITL) learning has been introduced as a means to address this issue by incorporating human feedback into the training process, allowing for real-time corrections and guidance in complex, high-risk driving scenarios [ 6 , 4 , 19 , 24 ] . A key approach within HITL systems is Imitation Learning (IL), which seeks to learn driving policies from expert demonstrations. Among IL methods, Behavioral Cloning (BC) [ 12 ] presents the most straightforward approach, using pre-collected expert demonstrations to train agents directly. However, the static nature of BC limits its adaptability, as it cannot incorporate human feedback to correct agent errors during deployment or guide learning during training. To address these challenges, Deep Q-Learning from Demonstrations (DQfD) [ 25 ] leverages expert demonstrations for pretraining, similar to Behavioral Cloning (BC), but further fine-tunes the policy using reinforcement learning. This approach retains the benefits of supervised learning while allowing the agent to improve beyond the limitations of static demonstrations through interaction with the environment. Meanwhile, HG-DAgger [ 13 ] introduces a more interactive approach by incorporating expert corrections iteratively during training, ensuring that the agent continuously refines its policy based on real-time human interventions. While progress has been made in integrating human input into DRL, a notable gap exists in effectively integrating human-agent collaboration. Existing research has often positioned the human as a supervisor, primarily emphasizing correction over active guidance or providing demonstrations in an offline setting, limiting real-time adaptability. This constraint limits the development of a truly collaborative approach, where human and agent inputs dynamically influence training and decision-making. Moreover, current approaches also lack a mechanism to validate the effectiveness of human interventions compared to agent-only decisions. To address these gaps, we propose the Interactive Double Deep Q-Network (iDDQN), an interactive DRL framework based on Clipped Double DQN [ 11 ] . iDDQN modifies the Q-value update equation to enable integration of real-time human interventions into the training process. This approach allows human and agent actions to be blended dynamically, fostering a collaborative policy that aligns with human intentions. An application domain where human interventions can significantly enhance decision-making is the dynamic and uncertain domain of autonomous driving. As demonstrated by Wu et al. [ 6 ] , real-time human guidance in DRL-based autonomous driving is valuable for enabling agents to adapt to unexpected obstacles and environmental variations. Building upon this insight, we evaluate our proposed iDDQN approach using the AirSim simulator [ 17 ] , a high-fidelity simulator that replicates complex driving conditions and facilitates real-time human interventions 1 1 1 While our primary evaluation focuses on the domain of autonomous driving, the iDDQN approach is designed for broader applicability across any domain that can be formulated as a Markov Decision Process (MDP), extending its relevance to various Human-in-the-Loop (HITL) scenarios. . Additionally, we propose an evaluative framework for post-hoc assessment of human interventions. This framework compares human inputs with agent-generated actions by estimating the cumulative rewards over a near-future horizon, providing a rigorous measure of the impact of human contributions. The key contributions of this work are as follows: 1. We propose the Interactive Double Deep Q-Network (iDDQN), a novel DRL method that merges Human-in-the-Loop interventions with agent decision-making, enabling the learning of unified, collaborative policies. 2. We further introduce an evaluative framework for qualitative comparison between human interventions and the agent’s potential output by comparing human and agent-generated actions based on estimated cumulative rewards. 3. Through extensive evaluations, we show that iDDQN outperforms HITL methods, including BC, DQfD, and HG-DAgger, achieving improved policy refinement, enhanced generalization, and faster convergence. Our results characterize the impact of human feedback on learning efficiency and policy performance, quantifying its benefits across different intervention strategies. II Related Work Recent advancements in Reinforcement Learning (RL) have significantly improved algorithmic efficiency and policy quality. The introduction of Deep Q-Learning (DQN) [ 8 ] marked a milestone by employing deep neural networks to approximate Q-value functions, enabling RL to handle high-dimensional state-action spaces. Building upon the foundation of DQN, Double DQN [ 7 ] resolved overestimation biases by decoupling action selection from evaluation, improving policy quality across various domains. The Dueling Architecture [ 3 ] improved learning by separating state-value and advantage-value functions, enhancing decision-making, particularly in vision-based scenarios. Clipped Double Q-Learning [ 11 ] enhanced stability in continuous action spaces, while Prioritized Experience Replay (PER) [ 9 ] increased sample efficiency by prioritizing transitions with higher learning potential. Human-in-the-Loop (HITL) methods have emerged as a critical component in enhancing RL for complex and dynamic environments. Behavioral Cloning (BC) [ 12 ] represents a foundational imitation learning approach that uses pre-collected expert demonstrations to train models via supervised learning. However, BC lacks adaptability for addressing errors during deployment. More interactive forms of Imitation Learning (IL), such as HG-DAgger [ 13 ] , incorporate expert interventions during training to correct catastrophic mistakes and iteratively refine policies through data aggregation. Similarly, Deep Q-Learning from Demonstrations (DQfD) [ 25 ] integrates human demonstrations into RL by pretraining on expert data, then fine-tuning with reinforcement learning while leveraging prioritized experience replay and an imitation loss. Interactive feedback mechanisms have been explored to bridge these gaps with surveys such as [ 1 ] highlight the diverse applications of HITL, including active learning [ 15 ] and real-time feedback [ 19 , 22 , 10 ] . Recent works, like [ 5 ] , focus on combining human feedback with simultaneous deployment, showcasing the potential of HITL in real-world settings. Strategies such as HACO [ 24 ] minimize the reliance on human intervention while ensuring safe agent behavior. Meanwhile, [ 6 ] demonstrated significant performance improvements by integrating real-time

Read the complete text