Uncertainty-Aware and Temporally Regulated Expert Advice in Reinforcement Learning for Autonomous Driving
Ahmed Abouelazm, Felix Klingebiel, Philip Schoerner, J. Marius Zöllner · 2026
Read Uncertainty-Aware and Temporally Regulated Expert Advice in Reinforcement Learning for Autonomous Driving by Ahmed Abouelazm, Felix Klingebiel, Philip Schoerner, J. Marius Zöllner (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30576.txt.
Introduction
Abouelazm and colleagues make reinforcement learning safer for self-driving cars by guiding exploration with expert advice only when uncertainty runs high. Advice triggers when epistemic or aleatoric uncertainty, gaps in knowledge or noise in the world, crosses thresholds that adapt as the agent gains confidence. A commitment-and-cooldown rule paces guidance so maneuvers stay coherent without draining the advice budget, and expert and agent experience share one replay buffer. In CARLA intersection tests, the method beat its baseline with higher success and fewer failures.
Catalog record
| CREATOR | Ahmed Abouelazm, Felix Klingebiel, Philip Schoerner, J. Marius Zöllner |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41495 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30576
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Uncertainty-Aware and Temporally Regulated Expert Advice in Reinforcement Learning for Autonomous Driving Ahmed Abouelazm, Felix Klingebiel, Philip Schoerner, J. Marius Zöllner Abstract Exploration in reinforcement learning for autonomous driving is inherently unsafe: agents must experience novel behaviors to learn, yet exploration can lead to collisions or off-road driving. We propose an uncertainty-aware framework that leverages expert advice to guide exploration while avoiding long-term dependence. Advice is triggered when epistemic or aleatoric uncertainty exceeds adaptive thresholds derived from rolling buffers, ensuring advice evolves with the agent’s confidence. A commitment–cooldown strategy with a stochastic early-stop heuristic regulates the duration and frequency of guidance, exposing the agent to coherent maneuvers without exhausting the advice budget. Expert and agent experiences are combined in a shared replay buffer within an off-policy implicit quantile network (IQN) backbone, enabling efficient reuse of expert trajectories. Experiments in CARLA show that our method outperforms the IQN baseline, improving success by 5–7% and reducing failures, demonstrating that risk-sensitive uncertainty coupled with regulated expert integration enables safer and more efficient exploration for sensor-based RL policy learning in unsignalized intersection navigation. I Introduction End-to-end (E2E) has emerged as a promising paradigm for autonomous driving (AD) , mapping raw sensory inputs directly to control actions and reducing reliance on handcrafted modular pipelines [ 6 ] . Among E2E approaches, reinforcement learning (RL) is particularly appealing: unlike Imitation learning (IL) , which depends on expert demonstrations, RL enables agents to adapt through direct interaction with the environment [ 32 ] . By exploring different behaviors and receiving feedback via rewards, RL agents can handle complex conditions such as occlusions, erratic drivers, and rare events that are underrepresented in datasets. However, RL also poses fundamental challenges in sample efficiency and safety [ 14 ] . The learning process is driven by the interplay between exploration, where the agent deliberately samples novel actions to acquire new knowledge about the environment, and exploitation, where it applies its current policy to maximize expected rewards [ 32 ] . Exploration is indispensable for avoiding suboptimal policies and uncovering effective driving strategies, yet it inherently involves unsafe or undesirable behaviors [ 36 ] . At the same time, extensive exploration leads to poor sample efficiency, as agents require a vast number of interactions before converging to a reliable policy [ 12 ] . These limitations underscore the need for mechanisms that constrain or prune exploration, directing the agent toward informative regions of the state space while preventing unnecessary behaviors. Research Gap. Existing work on safe exploration in RL for AD has followed several directions. Formal approaches such as Constrained Markov Decision Process (CMDP) [ 17 ] , and Lyapunov functions [ 25 ] encode safety explicitly, but they require handcrafted risk definitions, involve complex optimization, and provide no mechanism to guide policies toward safer actions. To address these limitations, expert knowledge has been introduced through demonstrations. Demonstration-based pre-training [ 7 ] accelerates convergence yet leaves agents vulnerable to unsafe exploration and provides no corrective feedback, while human-in-the-loop systems [ 18 ] allow targeted interventions but are not scalable and remain detached from the agent’s internal decision process. More scalable alternatives replace humans with rule-based [ 38 ] or learned [ 41 ] expert policies, which intervene automatically based on predefined heuristics. Although effective at preventing unsafe rollouts, rule-based triggers are overly conservative and context-dependent, while learned experts assume strong generalization and may overwrite valid actions with poor ones when this assumption fails. Recent advances shift control to the agent, allowing it to query expert advice when needed via triggers such as state uncertainty [ 8 ] and state novelty [ 19 ] . However, these approaches remain largely state-centric, neglect the risk induced by the agent’s exploratory actions, and lack mechanisms to regulate the frequency and impact of advice. To address these shortcomings, methods are needed that extend uncertainty estimation to capture action-related risk, combined with adaptive mechanisms that regulate advice and embed safety into the learned policy. Contribution. This work proposes an RL framework for AD that improves exploration safety by extending uncertainty estimation to account for action-related risk and integrating expert input through adaptive mechanisms. The key contributions are: • Uncertainty-aware expert guidance: the agent requests advice in states with high epistemic or aleatoric uncertainty, storing both expert and agent transitions in a shared buffer for training. • Uncertainty estimation: aleatoric uncertainty (environmental risk) is derived from return variance, while epistemic uncertainty (limited knowledge) is measured via bootstrapped ensembles, enabling detection of risky and underexplored states. • Commitment–cooldown strategy: regulates advice use by applying expert actions over short horizons and enforcing cooldowns, ensuring effective guidance without overreliance. II related work Safe RL in AD is challenging due to the need to balance efficiency with safety requirements. A common approach is to regulate exploration through constraints or learned objectives. CMDP formulations constrain cumulative risk below predefined thresholds [ 39 , 17 ] , while Lyapunov-based methods enforce stability by requiring a Lyapunov function to decrease along trajectories [ 25 , 11 ] . Control Barrier Functions (CBFs) define forward-invariant safe sets and enforce constraints to keep trajectories within these sets [ 33 , 34 ] , and distributional RL optimizes risk-sensitive returns such as Conditional Value at Risk (CVaR) [ 3 , 21 ] . Despite their differences, these approaches share key limitations: they solve complex optimization problems, depend on manually defined costs or stability conditions, or rely on learned objectives that are especially noisy at the start of training, and crucially, they lack mechanisms to actively guide the policy toward safer actions. This prompted the use of expert knowledge and demonstrations, which provide more direct and effective signals during exploration. Several works utilize demonstrations to pre-train policies, which accelerates convergence but leaves the agent vulnerable to unsafe exploration and provides no corrective feedback during training [ 7 , 31 ] . Early integration strategies alternate between collecting episodes from the expert and from the agent’s policy [ 13 , 5 ] , which improves robustness but still limits expert input to entire episodes rather than targeted interventions. As these episodes predominantly capture nominal driving and seldom illustrate recovery from unsafe states, these methods fail to teach agents how to act in the situations where guidance is most needed. This limitation has driven interest in more active systems, where experts can intervene, or the agent selectively queries advice. A common form of active expert intervention is human-in-the-loop training [ 30 , 18 ] , where a supervisor monitors the environment and provides corrective actions to guide the agent out of unsafe states. While effective, this approach is not scalable for long training, introduces bias in when and how interventions occur, and remains detached from the agent’s decision process, as humans cannot fully observe the agent’s intentions and may override potential recovery actions prematurely. These challenges have shifted attention toward alternative expert policies that can offer more scalable and consistent intervention during training. For such alternative policies, explicit intervention signals are needed to decide when to prune unsafe exploration. One direction relies on rule-based definitions of risk, where interventions occur if the risk associated with the agent’s behavior exceeds a threshold [ 27 , 21 ] or if the behavior