Finetuning Generative Trajectory Model with Reinforcement Learning from Human Feedback

Derun Li, Jianwei Ren, Yue Wang, Xin Wen, Pengxiang Li, Leimeng Xu, Kun Zhan, Zhongpu Xia, Peng Jia, Xianpeng Lang, Ningyi Xu, Hang Zhao · 2025

Read Finetuning Generative Trajectory Model with Reinforcement Learning from Human Feedback by Derun Li, Jianwei Ren, Yue Wang, Xin Wen, Pengxiang Li, Leimeng Xu, Kun Zhan, Zhongpu Xia, Peng Jia, Xianpeng Lang, Ningyi Xu, Hang Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10434.txt.

Introduction

Derun Li and co-authors fine tune a self driving path maker with human feedback so cars drive in a personal style, in a 2025 arXiv paper. Path makers trained on logs learn safe lines, yet they flatten the quirks that mark one driver's style from the next. Logs also bend toward dull cases.

Their frame, TrajHF, adds a two track refine step. A noise based maker offers many path shapes at once, and scored human likes then steer it by reward learning, past plain copying. Guard rails for safety and road sense stay in force. On a public driving bench the tuned maker matched the best known scores. The team pitches the frame as a new base for driving that feels like its owner.

Catalog record

CREATORDerun Li, Jianwei Ren, Yue Wang, Xin Wen, Pengxiang Li, Leimeng Xu, Kun Zhan, Zhongpu Xia, Peng Jia, Xianpeng Lang, Ningyi Xu, Hang Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-10434

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Finetuning Generative Trajectory Model with Reinforcement Learning from Human Feedback Derun Li, Jianwei Ren, Yue Wang, Xin Wen, Pengxiang Li, Leimeng Xu, Kun Zhan, Zhongpu Xia, Peng Jia, Xianpeng Lang, Ningyi Xu, Hang Zhao Abstract Generating human-like and adaptive trajectories is essential for autonomous driving in dynamic environments. While generative models have shown promise in synthesizing feasible trajectories, they often fail to capture the nuanced variability of human driving styles due to dataset biases and distributional shifts. To address this, we introduce TrajHF , a human feedback-driven finetuning framework for generative trajectory models, designed to align motion planning with diverse driving preferences. TrajHF incorporates multi-conditional denoiser and reinforcement learning with human feedback to refine multi-modal trajectory generation beyond conventional imitation learning. This enables better alignment with human driving preferences while maintaining safety and feasibility constraints. TrajHF achieves PDMS of 93.95 on NavSim benchmark, significantly exceeding other methods. TrajHF sets a new paradigm for personalized and adaptable trajectory generation in autonomous driving. 1 Introduction In autonomous driving, trajectory planning is responsible for generating safe, feasible, and human-like motions in complex and dynamic traffic environments. Heuristics-based learning methods [ 1 , 2 , 3 ] perform well in structured scenarios but struggle to generalize across diverse driving conditions. Imitation learning, most prominently generative models [ 4 , 5 , 6 , 7 ] , provides a data-driven alternative. They synthesize diverse human-like trajectories, but do not explicitly capture the nuanced preferences that characterize human driving behaviors, such as individual tendencies, regional norms, and environmental factors. Learning from demonstrations is fundamentally constrained by dataset bias. Behavior cloning only captures an average driving behavior, failing to represent the full spectrum of maneuvers. Generative imitation learning, on the other hand, learns real data distributions in the dataset, but is usually dominated by frequent modes, and hardly performs the best behavior unless sampling is conducted in an exhaustive manner. This misalignment degrades model performance in scenarios requiring deviations from frequent demonstrations, such as aggressive adaptations in complex traffic interactions, as shown in Fig 1 . Furthermore, human driving preferences are influenced not only by kinematic constraints but also by higher-level cognitive and social factors, such as risk tolerance, interactions with other road users, and situational awareness. Existing methods struggle to encode these latent factors, leading to planned motions that, while technically feasible, may feel unnatural, overly conservative, or unpredictably assertive. This further reduces user trust and acceptability, underscoring the need for trajectory generation methods that align with diverse human driving behaviors. To address this challenge, we propose TrajHF , a novel framework to finetune generative trajectory models using human feedback to align trajectory generation with diverse human driving preferences. Human feedback, in the form of comparative trajectory rankings or explicit annotations, provides a rich supervisory signal that extends beyond conventional imitation learning paradigms. By integrating this feedback into the finetuning process, our approach enables generative models to capture human-preferred driving styles while maintaining safety and feasibility constraints. We systematically investigate methodologies for incorporating human feedback into trajectory model finetuning. We propose a novel Multi-Conditioned Denoiser (MDC) Transformer network to generate multi-modal trajectory planning. Our design of conditioning multi-modal information gains significant improvement of trajectory generation quality, reaching PDMS 93.95 on NavSim [ 8 ] benchmark with uniform sampling and selection without any explicit guidance like anchors. Meanwhile, we explore reinforcement learning with human feedback and leverage group computation gathering rewards from each mode of generated trajectories to iteratively refine the generative trajectory distribution. Experimental results demonstrate that our method improves the “aggressiveness” of driving style of generated trajectories compared with the ground truth human preference data while preserving critical motion planning constraints. Using human feedback as an auxiliary learning signal, TrajHF offers a more personalized and adaptable trajectory generation paradigm, paving the way for autonomous vehicles that better reflect human driving behavior while ensuring safety and efficiency. The primary contributions of this paper are as follows: 1. We systematically investigate the trajectory distribution problem between the average behaviors and the human driving preferences. 2. We propose a human feedback-driven finetuning framework for generative trajectory models, enabling alignment with diverse human driving preferences. 3. Comprehensive experimental evaluations demonstrate the effectiveness of TrajHF in generating human-aligned autonomous driving trajectories across diverse scenarios. 4. TrajHF achieves SoTA on NavSim benchmark. 2 Related Work 2.1 End-to-end Trajectory Planning Imitation learning (IL) has been widely adopted for end-to-end trajectory planning by learning from expert demonstrations [ 9 ] . UniAD [ 10 ] and ViP3D [ 11 ] leverage bird-eye-view (BEV) representation and introduce a unified transformer structure to construct planning-centric frameworks. Likewise, VAD [ 12 ] and VADv2 [ 13 ] enhance this method by discretizing the trajectory space and creating a trajectory vocabulary, transforming the problem from regression to classification. [ 8 ] proposes NavSim, which establishes a new end-to-end autonomous driving planning benchmark focusing on multi-agent interactions and realistic driving scenarios. Transfuser [ 14 ] serves as a popular baseline on NavSim and a powerful encoder module, by fusing perceptual information from lidar and image data. FusionAD [ 15 ] took this fusion approach a step further by transmitting the combined perception features directly to the prediction and planning modules. Hydra-MDP [ 16 ] introduces a trajectory vocabulary for multi-target hydra-distillation, allowing the model to learn from both rule-based planners and human drivers in a scalable manner. In contrast, our end-to-end diffusion policy for planning does not utilize any form of guide information such as anchors or vocabulary, which directly generates continuous, diverse trajectories without being constrained by predefined discretized spaces, allowing for more multi-modal, accurate and robust planning in complex environments. 2.2 Generative Trajectory Model Generative models have emerged as a promising approach for trajectory prediction or planning, offering the ability to capture multi-modal driving behaviors. Early works apply Generative Adversarial Networks (GANs) [ 17 , 18 ] and Variational Autoencoders (VAEs) [ 19 , 20 ] to learn trajectory distributions, while more recent advances leverage autoregressive models [ 7 ] and diffusion models to generate diverse and realistic motion patterns [ 4 , 5 ] . In end-to-end autonomous driving, VBD [ 21 ] introduces a combination of denoising and behavior prediction losses alongside a query-centric Transformer encoder. DiffusionDrive [ 22 ] uses truncated diffusion policy to address the issues of mode collapse and heavy computational overhead. GoalFlow [ 3 ] integrates flow matching with goal point guidance for trajectory generation. Previous generative models focus mainly on ensuring safe and feasible motion generation by optimizing towards ground truth but lack mechanisms to incorporate the variability and complexity of human preferences, making them less effective in creating truly human-aligned autonomous driving systems, which is considered in our framework TrajHF . 2.3 Finetuning with Human Feedback Reinforcement Learning from Human Feedback (RLHF) [ 23 ] has emerged as a pivotal approach for incorporating human preferences into model training through reinforcement learning. This framework has demonstrated remarkable efficacy across various domains [ 24 , 25 , 26 , 27 ] . The fundamental principle underlying RLHF involves constructing a preference-based loss from limited human feedback data, subsequently employing reinforcement learning to finetune models toward the preference distribution while preserving their original capabilities. [ 28 ] pioneered the application of

Read the complete text