CRLLK: Constrained Reinforcement Learning for Lane Keeping in Autonomous Driving Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA 23

Xinwei Gao, Arambam James Singh, Gangadhar Royyuru, Michael Yuhas, Arvind Easwaran · 2025

Read CRLLK: Constrained Reinforcement Learning for Lane Keeping in Autonomous Driving Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA 23 by Xinwei Gao, Arambam James Singh, Gangadhar Royyuru, Michael Yuhas, Arvind Easwaran (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22248.txt.

Introduction

Xinwei Gao and four co-authors teach a car to keep its lane with constrained reinforcement learning, in work from 2025. Lane keeping balances competing goals, and hand-tuning their weights per scenario is brittle. Their setup folds the weights into the learning itself, so the agent learns the trade-offs alongside the driving policy. Tests beat standard reinforcement learning on efficiency and reliability, and real-vehicle demonstrations are reported.

Catalog record

CREATORXinwei Gao, Arambam James Singh, Gangadhar Royyuru, Michael Yuhas, Arvind Easwaran
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE11 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22248

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CRLLK: Constrained Reinforcement Learning for Lane Keeping in Autonomous Driving Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA 23 Xinwei Gao, Arambam James Singh, Gangadhar Royyuru, Michael Yuhas, Arvind Easwaran Abstract . Lane keeping in autonomous driving systems requires scenario-specific weight tuning for different objectives. We formulate lane-keeping as a constrained reinforcement learning problem, where weight coefficients are automatically learned along with the policy, eliminating the need for scenario-specific tuning. Empirically, our approach outperforms traditional RL in efficiency and reliability. Additionally, real-world demonstrations validate its practical value for real-world autonomous driving. Y. Vorobeychik, S. Das, A. Nowé (eds.) Demonstration Track 1. Introduction (a) Langrangian multiplier \lambda_{1} (b) Average lane deviation J_{c_{\mathrm{lane}}}^{\pi_{\theta}} (c) Total distance reward J_{R}^{\pi_{\theta}} (d) Action distribution in evaluation. Figure 1 . (a-c) Learning curves for different constraint threshold values ( \alpha_{1} ). (d) Action distribution for one test episode. The problem of lane keeping (LK) is an instance of a challenging real-time sequential decision-making problem in the domain of self-driving cars or autonomous driving systems ( Chen et al. 2020 ; Netto et al. 2004 ; Sallab et al. 2016 ) . Traditional model-free reinforcement learning (RL) based approaches to the LK problem face the challenge of defining a reward function that manages trade-offs between multiple objectives ( Claussmann et al. 2019 ) . Prior RL approaches use fixed-weighted combinations for objectives such as driving distance ( Almási et al. 2020 ) , minimizing yaw angle ( Feher et al. 2018 ; Kővári et al. 2020 ) , crash avoidance ( Ashwin and Naveen Raj 2023 ; Yuan et al. 2020 ) , and lateral/longitudinal control ( Duan et al. 2020 ; Muhammad et al. 2020 ) . Traditional multi-objective reinforcement learning (MORL) approaches ( Van Moffaert and Nowé 2014 ; Hayes et al. 2022 ) tackle such problems by learning a set of optimal policies or applying scalarized reward schemes that rely on static weighting. However, a fundamental limitation of such approaches is that the weight coefficients are typically obtained using scenario-specific tuning and extensive grid searches, which is both time-consuming and computationally expensive for high-fidelity physical simulators. In response to this challenge, this work introduces a constrained reinforcement learning based formulation and learning approach for the LK problem, which dynamically adjusts the weight coefficients of different objectives. By leveraging this constrained formulation, our system significantly outperforms traditional approaches in terms of efficiency (travel distance) and reliability (lane deviations and avoidance of collisions). We present the weight coefficient learning process and validate our framework in both simulation and real-world settings. 1 1 1 Demonstration video: youtu.be/1BlwJOIUaGM 2 2 2 Source code: github.com/CPS-research-group/CPS-NTU-Public/tree/AAMAS2025 2. Constrained RL for Lane Keeping 2.1. Problem Formulation Lane keeping is considered as a multi-objective problem with reward and cost functions defined at each time step. The travel distance reward is r(s_{t},a_{t})=\mathrm{d}^{\mathrm{trv}} , where \mathrm{d}^{\mathrm{trv}} denotes the forward distance. The lane deviation cost , c_{\mathrm{lane}}(s_{t},a_{t})=\mathrm{d}^{\mathrm{lane}} , penalizes horizontal deviation from the lane center, with the average performance given by J_{c_{\mathrm{lane}}}^{\pi_{\theta}}=\mathbb{E}\big[\frac{1}{H}\sum_{t=0}^{H-1}c_{\mathrm{lane}}(s_{t},a_{t})\big] . The collision cost , c_{\mathrm{coll}}(s_{t},a_{t})=\mathbf{1}^{\mathrm{col}} , takes the value 1 if the agent collides with obstacles or boundaries, with performance J_{c_{\mathrm{coll}}}^{\pi_{\theta}}=\mathbb{E}\big[\sum_{t=0}^{H-1}c_{\mathrm{coll}}(s_{t},a_{t})\big] across multiple episodes. To ensure safe and efficient driving, we formulate lane-keeping as a constrained optimization problem: (1) \displaystyle\max_{\pi_{\theta}\in\Pi}J_{R}^{\pi_{\theta}},\;\;\text{s.t. }\;\;J_{c_{\mathrm{lane}}}^{\pi_{\theta}}\leq\alpha_{1},\;J_{c_{\mathrm{coll}}}^{\pi_{\theta}}\leq\alpha_{2}, where \alpha_{1} and \alpha_{2} are non-negative thresholds for costs. \alpha_{1} corresponds to a real-world distance in decimeters ( 10^{-1}m ). We apply Lagrangian relaxation (LR) technique ( Bertesekas 1999 ) to convert the constrained optimization in Equation ( 1 ) into an equivalent unconstrained optimization problem as follows: (2) \displaystyle\min_{\lambda_{i}\geq 0}\max_{\theta}L(\lambda_{1},\lambda_{2},\theta)=\min_{\lambda_{i}\geq 0}\max_{\theta}\Big[J_{R}^{\pi_{\theta}}-\sum_{i=1}^{2}\lambda_{i}\left(J_{C_{i}}^{\pi_{\theta}}-\alpha_{i}\right)\Big], where i\in\{1,2\} , L is Lagrangian, \lambda_{i} are Lagrangian multipliers for lane and collision costs. \lambda_{i} and \theta are updated following the gradient descent and ascent approach as follows: (3) \displaystyle\lambda_{i}^{\mathrm{new}} \displaystyle=\max\left(0,\lambda_{i}^{\mathrm{old}}-\eta_{i}\left(J_{C_{i}}^{\pi_{\theta}}-\alpha_{i}\right)\right),\quad i\in\{1,2\}, (4) \displaystyle\theta^{\mathrm{new}} \displaystyle=\theta^{\mathrm{old}}+\eta_{3}\nabla_{\theta}\mathbb{E}\Big[\log\pi_{\theta}(a|s)\Big(J_{R}^{\pi_{\theta}}-\lambda_{1}J_{C_{\mathrm{lane}}}^{\pi_{\theta}}-\lambda_{2}J_{C_{\mathrm{coll}}}^{\pi_{\theta}}\Big)\Big], where \eta_{1} , \eta_{2} and \eta_{3} are learning rates. Equations ( 3 ) updates \lambda_{i} to satisfy the original cost constraints. The policy parameters \theta is updated with the discounted cost version of J_{c_{\mathrm{lane}}}^{\pi_{\theta}} and J_{c_{\mathrm{coll}}}^{\pi_{\theta}} to track the recursive property of Bellman equation ( Bellman 1952 ) , where J_{c_{\mathrm{lane}\cdot\gamma}}^{\pi_{\theta}}\triangleq\mathbb{E}\left[\sum_{t=0}^{H-1}\gamma^{t}c_{\mathrm{lane}}(s_{t},a_{t})\right] and J_{c_{\mathrm{coll}\cdot\gamma}}^{\pi_{\theta}}\triangleq\mathbb{E}\left[\sum_{t=0}^{H-1}\gamma^{t}c_{\mathrm{coll}}(s_{t},a_{t})\right] . Using the discounted cost constraints above, we define a modified reward function that applied in our approach: (5) \displaystyle\hskip-3.41418pt\hat{r}(s_{t},a_{t},\lambda_{1},\lambda_{2})\triangleq r(s_{t},a_{t})-\lambda_{1}c_{\mathrm{lane}}(s_{t},a_{t})-\lambda_{2}c_{\mathrm{coll}}(s_{t},a_{t}). In the reward function \hat{r}(\cdot) , parameters \lambda_{1} and \lambda_{2} act as the adaptive weight coefficients of objectives. 2.2. Implementation Existing constraint RL solutions like RCPO ( Tessler et al. 2019 ) update policy with truncated samples, failing to track collision cost constraints across multiple episodes, while two-timescale frameworks are impracticable for high-fidelity physical simulators due to computational costs. We adopt a one-timescale framework, updating both \theta and (\lambda_{1},\lambda_{2}) simultaneously, akin to the simplifications of Actor-Critic ( Konda and Tsitsiklis 1999 ) made by A3C ( Mnih et al. 2016 ) and DDPG ( Silver et al. 2014 ) . We implement this approach on the Duckietown platform with PPO ( Schulman et al. 2017 ) algorithm. The system observes the vehicle’s state through camera and computes actions while minimizing lane deviation and avoiding obstacles. 3. Experiment and Demonstration 3.1. Simulation Evaluation The simulation evaluation results on a small loop scenario are listed in Table 1 . CRLLK-D, CRLLK-C are implementations of our approach on discrete and continuous setting, while PPO-D, PPO-C are the traditional approach with a fixed reward in ( Schulman et al. 2017 ) . Each method is evaluated on 100 test episodes and averaged over two different seeds. Further evaluation results for baselines and scenarios are listed in the video. This evaluation result shows significant performance improvement by adapting our constrained RL framework in terms of efficiency (higher J_{R} for travel distance), and reliability (lower J_{c_{\mathrm{lane}}} for lane deviations and lower J_{c_{\mathrm{coll}}} for collision). Table 1 . Performance comparison on the small loop scenario with \alpha_{1}=0.5 and \alpha_{2}=0.02 . CRLLK-D PPO-D CRLLK-C PPO-C J_{c_{\mathrm{lane}}} 0.66 \pm 0.02 0.99 \pm 0.07 0.31 \pm 0.00 1.04 \pm 0.25 J_{c_{\mathrm{coll}}} 0.17 \pm 0.00 0.38 \pm 0.04 0.05 \pm 0.01 0.34 \pm 0.20 J_{R} 69.0 \pm 0.6 46.6 \pm 3.2 62.4 \pm 22.2 51.3 \pm 14.8 (a) Small loop (b) Zig-zag (c) Obstacle loop Figure 2 . Real-world evaluation in scenarios. We performed a demonstration on the training convergence and testing behavior for CRLLK-D on a small loop map in Figure 1 , with two different constraint levels: tight constraint ( \alpha_{1}=0.5 ) and loosened constraint ( \alpha_{1}=0.75 ). For tight constraint, the agent policy has a higher probability to choose turn-left and turn-right actions than in the loose constraint, showing fewer lane deviations but covering a shorter total distance. Through this study, we gain insight into how weight coefficients are dynamically learned and how different constraint thresholds affect the lane-keeping trade-off between the lane deviation and travel distance. 3.2. Real-World Demonstration We evaluate the constrained RL-based policy for lane-keeping tasks using a Duckiebot in real-world scenarios after training in simulation. Testing covered three scenarios: low-difficulty ( small loop ), high-difficulty ( zig-zag ), and complex ( obstacle loop ). The Duckiebot, equipped with Jetson Nano ( Süzen et al. 2020 ) hardware and ROS2 ( Macenski et al. 2022 ) , uses an

Read the complete text