MTA-RL: Robust Urban Driving via Multi-modal Transformer-based 3D Affordances and Reinforcement Learning Guangli Chen, Dianzhao Li, Wenjian Zhong, Bangquan Xie, Ostap Okhrin Abstract Robust urban autonomous driving requires reliable 3D scene understanding and stable decision-making under dense interactions. However, existing end-to-end models lack interpretability, while modular pipelines suffer from error propagation across brittle interfaces. This paper proposes MTA-RL , the first framework that bridges perception and control through M ulti-modal T ransformer-based 3D A ffordances and R einforcement L earning (RL). Unlike previous fusion models that directly regress actions, RGB images and LiDAR point clouds are fused using a transformer architecture to predict explicit, geometry-aware affordance representations. These structured representations serve as a compact observation space, enabling the RL policy to operate purely on predicted driving semantics, which significantly improves sample efficiency and stability. Extensive evaluations in CARLA Town01–03 across varying densities (20–60 background vehicles) show that MTA-RL consistently outperforms state-of-the-art baselines. Trained solely on Town03, our method demonstrates superior zero-shot generalization in unseen towns, achieving up to a 9.0% increase in Route Completion, an 11.0% increase in Total Distance, and an 83.7% improvement in Distance Per Violation. Furthermore, ablation studies confirm that our multi-modal fusion and reward shaping are critical, significantly outperforming image-only and unshaped variants, demonstrating the effectiveness of MTA-RL for robust urban autonomous driving. I INTRODUCTION Urban autonomous driving requires vehicles to operate under partial observability and uncertainty while simultaneously solving tightly coupled tasks such as navigation, trajectory planning, dynamic obstacle avoidance, traffic rule compliance, and safe interaction with vulnerable road users [ 30 , 21 , 32 , 37 , 16 , 14 , 36 ] . To address this challenge, two dominant paradigms have emerged: modular pipelines and end-to-end learning . While modular approaches [ 27 , 13 ] offer interpretability, they heavily rely on hand-crafted priors, and upstream errors can propagate downstream, degrading robustness in highly dynamic scenarios [ 18 ] . Alternatively, end-to-end learning aims to overcome these limitations by directly mapping sensory inputs to control commands [ 25 , 3 ] . Within this paradigm, imitation learning (IL) [ 12 ] is data-efficient but suffers from compounding errors caused by distribution shifts in long-tail scenarios. Unlike IL, RL [ 24 ] is capable of surpassing expert demonstrations, but it often exhibits poor sample efficiency and limited interpretability when processing high-dimensional raw observations. Moreover, by tightly coupling perception and decision-making, pure end-to-end RL lacks explicit spatial reasoning, making it difficult to interpret failures or guarantee safety in novel environments. To bridge the gap between modular pipelines and end-to-end learning, direct perception has been proposed as an intermediate paradigm [ 2 ] . Direct perception learns compact, task-relevant intermediate representations from raw sensory inputs, commonly referred to as affordances . These affordances encode essential driving information such as ego-vehicle states, traffic rules, and interactions with surrounding agents, enabling decision-making in a structured and semantically meaningful state space. By decoupling perception from control, affordance-based systems improve interpretability and robustness, and allow downstream control to be implemented using either rule-based [ 22 ] or learning-based policies [ 1 ] . Despite these advantages, most existing affordance learning approaches rely solely on monocular camera images, restricting the learned representations to the 2D image plane and limiting their geometric expressiveness. This weak 3D geometric reasoning leads to unstable policies in dense urban traffic. Overcoming this requires multi-modal fusion. While standard CNN-based fusion methods often suffer from inherent viewpoint misalignment, leading to depth and feature blurring across modal views, and incur significant geometric information loss when forcing sparse 3D points into grid-structured convolutions [ 29 , 17 ] , recent advances in transformer-based architectures [ 28 , 6 ] have demonstrated strong capabilities in multi-modal fusion, enabling effective integration of RGB images and LiDAR point clouds for richer 3D scene understanding [ 29 , 19 , 23 ] . However, how to leverage such multi-modal representations to support safe and efficient decision-making remains an open challenge. Motivated by these observations, we propose MTA-RL , the first unified framework that couples multi-modal 3D Bird’s-Eye-View (BEV) affordances with RL for urban driving. MTA-RL employs a transformer-based fusion network [ 26 ] to jointly process RGB images and LiDAR point clouds in BEV, producing compact, geometry-aware, and safety-critical 3D affordances. These learned affordances serve as the structured state representation for a dedicated RL controller, as illustrated in Fig. 1 . Operating in this semantically grounded space improves sample efficiency, interpretability, and robustness, while facilitating the integration of traffic rules and explicit safety constraints through reward shaping and termination conditions [ 22 , 34 ] . Our contributions are summarized as follows: • 3D BEV affordance : We propose a transformer-based multi-modal architecture with explicit distance-to-constraint heads to predict compact, geometry-aware affordances in BEV space. • RL operating purely on predicted affordances : The learned affordances provide a structured observation space, enabling stable RL policy learning in complex urban environments. • Robust urban driving performance : Extensive evaluations in CARLA demonstrate that MTA-RL consistently outperforms state-of-the-art baselines in both affordance prediction accuracy and driving performance across diverse towns and traffic densities. Fig. 1: Overview of the MTA-RL framework. A transformer-based affordance prediction network fuses RGB camera images and LiDAR BEV inputs to produce compact, geometry-aware affordance representations. These structured affordances serve as input to an RL controller, enabling stable decision-making. II RELATED WORKS II-A Reliable Driving Systems Urban driving is fundamentally constrained by the limitations of individual sensing modalities: camera-based perception provides rich semantic information but suffers from inaccurate depth estimation and sensitivity to adverse illumination, whereas LiDAR offers precise geometric measurements but lacks semantic and texture cues [ 31 ] . Consequently, multi-modal sensor fusion has become a critical component of modern AD systems. Early fusion methods relied on explicit geometric alignment, such as projecting LiDAR points into image space [ 29 ] or combining features using continuous convolutions [ 17 ] . Although effective under ideal calibration, these approaches are sensitive to alignment errors and typically aggregate information locally, limiting their ability to capture global cross-modal interactions. More recently, transformer-based architectures have significantly advanced multi-modal fusion by leveraging attention mechanisms to model long-range dependencies across modalities. Notably, TransFuser [ 4 ] employs self-attention to softly associate image and LiDAR features, substantially outperforming rigid geometric fusion approaches. However, most transformer-based fusion models are trained using IL and directly regress waypoints or control commands. Such formulations remain susceptible to causal confusion and distribution shift [ 5 ] , often resulting in fragile behavior in rare or safety-critical situations. To achieve robust performance in such critical scenarios, our framework explicitly decouples perception from decision-making to empower a subsequent RL policy. II-B Affordance Learning and Intermediate Representations Early affordance learning methods, such as DeepDriving [ 2 ] and CAL [ 22 ] , successfully extract task-relevant indicators (e.g., lane distances and traffic light states) to support downstream decision-making. However, these methods primarily rely on hand-crafted, rule-based controllers, which struggle to handle the complex, dynamic interactions typical of dense urban environments. To address this, more recent approaches like [ 1 ] and [ 26 ] replace rule-based systems with RL policies. Despite achieving improved adaptability, these methods dominantly rely on single-modal camera inputs, restricting their representations to the 2D image plane. The inherent lack of explicit 3D geometric reasoning severely limits their ability to accurately model multi-agent interactions. In contrast, we construct multi-modal affordances directly in the 3D BEV space to restore the geometric expressiveness required for robust downstream control. II-C Reinforcement Learning for Autonomous Driving While RL has shown promise for autonomous driving [ 30 , 11 ] , ensuring safety and sample efficiency in complex urban environments remains a fundamental challenge. To address safety, a recent work [ 33 ] augments the Soft Actor-Critic (SAC) algorithm [ 8 ] with model predictive control to enforce hard constraints; however, such methods are typically evaluated in structured expressway scenarios and struggle to scale to dense, unsignalized urban intersections. For urban navigation, Roach [ 35 ] employs a PPO policy, but it heavily relies on privileged simulator states, limiting its practical interpretability. More recently, world-model-based RL methods like Think2Drive [ 15 ] improve efficiency but lack the explicit state representations required to enforce strict traffic rules and geometric safety boundaries. Unlike these implicit or privileged baselines, MTA-RL operates purely on the predicted explicit affordances. Combined with dense reward shaping, this ensures strict semantic and geometric compliance in highly interactive urban traffic. III METHODOLOGY This section presents MTA-RL , a unified framework that couples multi-modal 3D affordance learning with RL for urban autonomous driving. MTA-RL first fuses heterogeneous sensor inputs to predict structured, safety-critical affordances in a geometry-aware BEV representation. These affordances then serve as the state space for an RL agent augmented with an explicit, rule-aware safety wrapper that enforces traffic regulations and interaction safety. III-A Transformer-Based Multimodal Affordance Prediction To enable robust decision-making, we adopt a transformer-based fusion architecture [ 4 ] to integrate RGB and LiDAR inputs. These fused multi-modal features are subsequently utilized to predict structured, geometry-aware driving affordances. III-A 1 Multimodal Sensor Fusion The proposed framework leverages two complementary sensing modalities: RGB camera images and LiDAR point clouds. For visual perception, three forward-facing RGB cameras (left 60^{\circ} , center, right 60^{\circ} ) are employed, each with a horizontal field of view (FOV) of 120^{\circ} . After distortion correction and cropping, the images are concatenated into a composite image of resolution 704\times 160 , covering an effective FOV of 132^{\circ} . This representation captures rich semantic cues such as traffic lights, pedestrians, and surrounding vehicles, and is processed using a ResNet-34 backbone [ 9 ] to extract multi-scale image features. For geometric perception, LiDAR point clouds are projected onto a 32\,\mathrm{m}\times 32\,\mathrm{m} BEV grid centered on the ego vehicle, with a spatial resolution of 256\times 256 ( 0.125\,\mathrm{m} per cell). Points are discretized into two height-based channels corresponding to ground and obstacle returns. Additionally, the 2D navigation goal is rasterized and embedded as a third channel, forming a three-channel BEV representation. This design introduces early spatial alignment between geometry-aware perception and route guidance. Following fusion, the multi-scale feature maps are aggregated into a single latent vector \mathbf{z} via global average pooling. The latent vector \mathbf{z} encodes global scene context, cross-modal interactions, and navigation intent. It serves as a shared input to all affordance prediction heads (Section III-A2 ), enabling consistent perception outputs and providing a compact, semantically structured representation for downstream policy learning. III-A 2 Affordance Prediction To extract task-relevant driving semantics from the shared latent representation \mathbf{z} , we employ a multi-task prediction architecture with task-specific heads. Each head is implemented as a lightweight multi-layer perceptron (MLP) and predicts a single driving affordance. All heads operate in parallel on \mathbf{z} , forming a one-to-many prediction structure that promotes efficient feature sharing and joint optimization across tasks. All affordances are predicted in the ego-centric BEV coordinate frame and are expressed as \mathbf{a}_{j}=\mathrm{MLP}_{j}(\mathbf{z}) , where \mathbf{a}_{j} denotes the prediction of the j -th driving affordance. Based on their functional roles, we categorize the predicted affordances into discrete (categorical) and continuous (regression) affordances. Discrete affordances Discrete affordances encode high-level semantic constraints associated with traffic rules and interaction safety: Red traffic light and Stop sign indicate active regulatory constraints, and Hazard stop signals imminent collision risks from pedestrians or vehicles. Continuous affordances Continuous affordances capture geometric and relational information required for precise longitudinal and lateral control: These explicitly include: Lateral distance and relative angle , representing the ego vehicle’s offset and heading deviation from the lane centerline; Target vehicle distance , measuring the longitudinal gap to the leading vehicle; as well as Traffic light distance , Stop sign distance , and Pedestrian distance , which provide the continuous longitudinal distances to the respective semantic entities ahead. III-B Perception-Guided Reinforcement Learning Policy Based on the predicted driving affordances from the perception module, we formulate high-level driving decision-making as an RL problem. We adopt SAC as the policy learning algorithm due to its stability and sample efficiency in continuous control settings. The following section details the RL formulation and training setup. III-B 1 Action Space and Observation Space At each timestep t , the agent outputs a continuous action vector a(t)=(a_{\text{steer}}(t),a_{\text{long}}(t))\in[-1,1]^{2} . Here, a_{\text{steer}} controls the lateral steering, and a_{\text{long}} specifies the longitudinal command, mapping positive and negative values to throttle and braking, respectively. The observation o(t)=(o^{\text{aff}}(t)^{\top},o^{\text{ego}}(t)^{\top})^{\top} concatenates the perception-derived affordances and the ego-vehicle state. The affordance component is defined as: o^{\text{aff}}(t)=(d_{\text{lat}},\theta,d_{\text{veh}},d_{\text{tl}},d_{\text{stop}},d_{\text{ped}},I_{\text{red}},I_{\text{stop}},I_{\text{hazard}})_{t}^{\top}, where d_{\text{lat}} and \theta denote the lateral offset and relative heading. The variables d_{\text{veh}},d_{\text{tl}},d_{\text{stop}} , and d_{\text{ped}} measure longitudinal distances to the leading vehicle, traffic light, stop sign, and pedestrian, respectively. The binary indicators I_{\text{red}},I_{\text{stop}} , and I_{\text{hazard}} indicate the presence of an active red traffic light, a stop-sign constraint, and imminent collision hazards induced by vehicles or pedestrians, respectively. The ego-state component captures the current speed v(t) , as well as the steering and longitudinal commands from the previous time step: o^{\text{ego}}(t)=(v(t),a_{\text{steer}}(t-1),a_{\text{long}}(t-1))^{\top}. By explicitly separating semantics from vehicle dynamics, this decoupled formulation yields a compact representation that facilitates stable policy learning. III-B 2 Reward Function The reward function jointly optimizes multiple objectives to promote robust and safe driving behavior in complex urban environments. At each time step t , the overall reward R(t) is defined as: \displaystyle R(t) \displaystyle=R_{\text{lane}}(t)+R_{\text{speed}}(t)+R_{\text{smooth}}(t) (1) \displaystyle+R_{\text{prog}}(t)+R_{\text{rule}}(t)+R_{\text{event}}(t). To achieve an optimal balance among these diverse driving objectives, the specific weighting coefficients for each component were determined through extensive empirical evaluations. These terms respectively regulate lane-keeping, speed tracking, control smoothness, route progress, traffic rule compliance, and safety-critical terminal events, as detailed below. Lane Keeping The lane-keeping reward penalizes both lateral offset and misalignment through smooth, bounded shaping terms: R_{\text{lane}}(t)=w_{\ell 1}\cdot\exp\!\left(-2.5\,\lvert d_{\text{lat}}(t)\rvert\right)+w_{\ell 2}\cdot\frac{1}{1+3\,\lvert\theta(t)\rvert}, where w_{\ell 1} and w_{\ell 2} are set to 1.0 and 0.2 , respectively. Speed Tracking To promote efficiency and penalize overspeeding, we define: \displaystyle R_{\text{speed}}(t) \displaystyle=w_{s1}\cdot\Biggl(1-\min\!\Bigl(1,\,\frac{\lvert v(t)-v_{\text{des}}(t)\rvert}{v_{\max}}\Bigr)\Biggr) \displaystyle+w_{s2}\cdot\,\frac{\max\!\bigl(0,\,v(t)-v_{\max}\bigr)}{v_{\max}}, where w_{s1}=0.65 and w_{s2}=-0.7 are fixed scalar weights, v(t) denotes the current ego-vehicle speed, v_{\text{des}}(t) is the desired speed, and v_{\max} is the maximum allowable speed. Control Smoothness To discourage abrupt actions and oscillatory behavior while promoting comfortable driving, we define a control smoothness term based on the step-to-step change in the steering command a_{\text{steer}}(t) and the variation of the heading error \theta(t) relative to the lane centerline: \displaystyle R_{\text{smooth}}(t)={} \displaystyle w_{sm1}\cdot\,|a_{\text{steer}}(t)-a_{\text{steer}}(t-1)|+ \displaystyle w_{sm2}\cdot\,|\theta(t)-\theta(t-1)|, where w_{sm1}=-0.7 and w_{sm2}=-0.5 . Route progress reward The progress reward is: R_{\text{prog}}(t)=w_{pr1}\cdot\Bigl(\Delta k(t)-\mathbb{I}_{\text{free}}(t)\bigl(1-\mathbb{I}_{\text{move}}(t)\bigr)\Bigr), where w_{pr1} is set to 0.4 , \Delta k(t) denotes the waypoint index increment. \mathbb{I}_{\text{move}}(t)=\mathbb{I}\!\left[\Delta k(t)>0\right] is indicator function, showing whether the vehicle is making progress. \mathbb{I}_{\text{free}}(t) , on the other hand, shows if the stopping constraint is triggered. Traffic Rule Compliance Shaping To enforce safe and anticipatory stopping under traffic regulations, including traffic lights, stop signs, and pedestrian yielding, we adopt a unified deceleration shaping strategy. Let d(t) denote the longitudinal distance from the ego vehicle to the active constraint, the traffic rule compliance shaping reward is defined as: \displaystyle R_{\text{rule}}(t)= \displaystyle w_{r1}\cdot\,\eta\left(d(t)\right)\cdot\frac{v(t)}{v_{\max}}+ \displaystyle w_{r2}\cdot\mathbb{I}\!\left[d(t)<2\wedge v(t)