Beyond Scalar Rewards: Distributional Reinforcement Learning with Preordered Objectives for Safe and Reliable Autonomous Driving

Ahmed Abouelazm, Jonas Michel, Daniel Bogdoll, Philip Schörner, J. Marius Zöllner · 2026

Read Beyond Scalar Rewards: Distributional Reinforcement Learning with Preordered Objectives for Safe and Reliable Autonomous Driving by Ahmed Abouelazm, Jonas Michel, Daniel Bogdoll, Philip Schörner, J. Marius Zöllner (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20230.txt.

Introduction

Abouelazm and co-authors give self driving reinforcement learning a ranked set of goals instead of one blended score. Safety, efficiency, and comfort often conflict on the road. Weighted sums hide those priorities and can yield unsafe policies. Their preordered framework keeps a hierarchy over reward parts. A new comparison, quantile dominance, judges action return distributions without collapsing them. An algorithm then keeps the actions that stay non dominated per goal. Built on implicit quantile networks, tests in Carla showed fewer collisions and better success rates.

Catalog record

CREATORAhmed Abouelazm, Jonas Michel, Daniel Bogdoll, Philip Schörner, J. Marius Zöllner
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36904
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20230

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Scalar Rewards: Distributional Reinforcement Learning with Preordered Objectives for Safe and Reliable Autonomous Driving Ahmed Abouelazm, Jonas Michel, Daniel Bogdoll, Philip Schörner, J. Marius Zöllner Abstract Autonomous driving involves multiple, often conflicting objectives such as safety, efficiency, and comfort. In reinforcement learning (RL), these objectives are typically combined through weighted summation, which collapses their relative priorities and often yields policies that violate safety-critical constraints. To overcome this limitation, we introduce the Preordered Multi-Objective MDP (Pr-MOMDP), which augments standard MOMDPs with a preorder over reward components. This structure enables reasoning about actions with respect to a hierarchy of objectives rather than a scalar signal. To make this structure actionable, we extend distributional RL with a novel pairwise comparison metric, Quantile Dominance (QD), that evaluates action return distributions without reducing them into a single statistic. Building on QD, we propose an algorithm for extracting optimal subsets, the subset of actions that remain non-dominated under each objective, which allows precedence information to shape both decision-making and training targets. Our framework is instantiated with Implicit Quantile Networks (IQN), establishing a concrete implementation while preserving compatibility with a broad class of distributional RL methods. Experiments in Carla show improved success rates, fewer collisions and off-road events, and deliver statistically more robust policies than IQN and ensemble-IQN baselines. By ensuring policies respect rewards preorder, our work advances safer, more reliable autonomous driving systems. I Introduction End-to-End (E2E) learning has emerged as a compelling paradigm for Autonomous Driving (AD) , directly mapping raw sensory input to vehicle actions within a unified model [ 35 ] . By bypassing handcrafted intermediate stages, E2E approaches mitigate error accumulation and enable scalable, data-driven driving policies [ 11 ] . Unlike modular pipelines, which rely on carefully engineered perception and decision-making components, E2E systems reduce error propagation between modules and allow joint optimization of perception and control. This streamlines learning and lowers reliance on manual design [ 18 ] . While imitation learning (IL) is effective for acquiring basic driving skills, Reinforcement Learning (RL) offers distinct advantages by optimizing behavior through direct interaction with the environment [ 29 ] . By maximizing cumulative rewards, RL agents can adapt to the dynamic and uncertain conditions of real-world traffic [ 24 ] . At the core of this process lies the reward function, which specifies driving objectives such as safety, efficiency, and comfort, and thus directly governs the quality of the learned policy [ 9 ] . Research Gap. Designing reward functions for AD is inherently complex, as they must capture multiple, often conflicting objectives, such as safety, efficiency, and comfort, while also reflecting their relative priorities [ 2 ] . A common practice is to collapse these objectives into a single scalar reward, typically through naive or weighted summation [ 24 ] . However, such formulations are difficult to tune, highly context-dependent, and have been shown to produce undesirable behaviors when trade-offs are misaligned, such as prioritizing comfort or efficiency at the expense of safety [ 25 ] . To address these deficiencies, research in Multi-Objective Reinforcement Learning (MORL) has proposed representing a reward as a vector of distinct objectives [ 13 ] . This allows the agent to learn separate value estimates per objective. However, decision-making still relies on weighted aggregation of these estimates [ 23 ] , which limits the learned policy’s ability to preserve the intended priority of objectives. Alternative approaches introduce hierarchical rewards inspired by rulebooks [ 8 ] , where relations among objectives are explicitly encoded [ 6 , 1 ] . While these approaches provide interpretability and structured priorities at the reward-design level, agents are still trained with scalar rewards, restricting agents’ ability to disentangle the contribution of each objective. Therefore, a gap remains in developing RL agents that can semantically represent multiple objectives and enforce their relative priorities within the learning process itself. Such a formulation enables safety-critical objectives to guide both learning and decision-making more reliably, leading to safer driving behavior. Contribution. This paper presents a framework that incorporates preorder relations, capturing the relative priority between objectives, directly into RL agents. In this way, objective priorities are respected during both training and inference. The key contributions of this work are: • Preordered Multi-Objective MDP (Pr-MOMDP): We extend MOMDPs with preorder relations, providing a formulation that enables reasoning about actions with respect to prioritized objectives. • Quantile-based action comparison: Building on the Pr-MOMDP formulation, we propose Quantile Dominance (QD), a distributional metric that compares full return distributions to derive pairwise action relations. • Optimal subsets for decision-making: Leveraging QD, we extract optimal action subsets, non-dominated actions under each objective, and integrate them into both action selection and training updates, ensuring higher-priority objectives consistently shape policy learning. II related work The integration of multi-objective and hierarchical reward structures into RL policies is a critical area of research. While significant progress has been made, major challenges remain. Experiments in the complex domain of AD have revealed that current approaches insufficiently respect reward structures, leading to undesirable behavior [ 25 ] . In this section, we first introduce classical reward structures and their challenges in the context of AD , and subsequently introduce prior work in MORL and Hierarchical Reinforcement Learning (HRL) that aims to handle such complex reward structures. II-A Reward Design in Autonomous Driving AD is a complex domain with a multitude of often conflicting objectives [ 2 ] . This makes it challenging to manually design reward functions and can often lead to insufficient performance [ 32 ] . Knox et al. [ 25 ] identified several challenges in the manual design of reward functions, such as undesired risk tolerance or preference orderings that do not align with human judgment. These findings are confirmed by a large-scale survey by Abouelazm et al. [ 2 ] , highlighting that most reward functions in the literature utilize different individual reward terms but aggregate them into a scalar output, eliminating context awareness and relative ordering. To avoid error-prone manual reward designs, another line of work proposes the automated generation of a reward function based on Large Language Models (LLMs) [ 16 ] . However, this approach still collapses multiple reward terms into a single scalar, not solving the core issue of classical reward structures. Finally, some works address the runtime adoption of driving behaviors by introducing priors on driving aspects, such as comfort or aggressiveness [ 22 , 33 ] . Here, classically engineered reward terms are combined with prior conditions so agents can display different behaviors during inference without re-training. These approaches emphasize certain aspects of total reward rather than integrating hierarchies to address complex reward structures. II-B Multi-Objective RL and Hierarchical Rewards. As classical reward structures struggle to address scenarios that require hierarchical or multi-objective reward signals, this section presents advancements in MORL and HRL . Rather than just decomposing components of the reward function, several works adapt model architectures by introducing multi-branch networks for individual reward components [ 37 , 40 , 27 , 23 , 21 , 30 ] . While these concepts have demonstrated performance improvements and can be used to dynamically adjust weights during runtime [ 20 , 5 ] , an artificial bottleneck is introduced by merging the resulting Q-values

Read the complete text