FlowR2A: Learning Reward-to-Action Distribution for Multimodal Driving Planning

Xirui Li Zhe Liu Xiaoqing Ye Wenhua Han, Yifeng Pan Junyu Han Hengshuang Zhao, project lead corresponding author https://lixirui142.github.io/flowr2a-ad · 2026

Read FlowR2A: Learning Reward-to-Action Distribution for Multimodal Driving Planning by Xirui Li Zhe Liu Xiaoqing Ye Wenhua Han, Yifeng Pan Junyu Han Hengshuang Zhao, project lead corresponding author https://lixirui142.github.io/flowr2a-ad (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24231.txt.

Introduction

Li and colleagues generate driving plans by asking a model what good outcomes look like, then sampling actions that promise them. Motion planning for self-driving cars splits between two camps. Scoring methods enjoy dense reward supervision but pick from a fixed menu of actions. Anchor methods propose freely but learn from only one recorded trajectory. Their 2026 model, FlowR2A, reframes rewards as generative conditions rather than grading targets.

A flow-matching decoder learns the distribution of actions conditioned on reward. It trains on dense pairs of trajectories and their simulated rewards. The model thus internalises how actions translate into safety, progress, comfort, and rule compliance. Fine-grained per-timestep conditioning, plus reward noise augmentation, balances hard safety constraints against softer progress goals. At test time, reward guidance and anchored sampling steer generation toward high-quality proposals. Li's design claims both camps' virtues in one generative planner.

Catalog record

CREATORXirui Li Zhe Liu Xiaoqing Ye Wenhua Han, Yifeng Pan Junyu Han Hengshuang Zhao, project lead corresponding author https://lixirui142.github.io/flowr2a-ad
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24231

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FlowR2A: Learning Reward-to-Action Distribution for Multimodal Driving Planning Xirui Li Zhe Liu Xiaoqing Ye Wenhua Han, Yifeng Pan Junyu Han Hengshuang Zhao, project lead corresponding author https://lixirui142.github.io/flowr2a-ad Abstract Multimodal driving planning faces a long-standing tension between two paradigms: scoring-based methods benefit from dense reward supervision but are confined to a fixed action vocabulary, while anchor-based methods generate proposals dynamically yet suffer from sparse supervision constrained to a single ground-truth trajectory. In this work, we propose FlowR2A, which resolves this tension by reframing simulation-based rewards from discriminative targets into generative conditions. By learning the reward-conditioned action distribution from dense trajectory-reward pairs with a flow-matching decoder, FlowR2A unifies the dense supervision of scoring-based methods with the proposal generation of anchor-based methods in a single generative model, forcing the model to internalize the correlation between an action and its outcomes in safety, progress, comfort, and rule compliance. To balance hard safety constraints against soft progress objectives, we introduce fine-grained per-timestep reward conditioning and reward noise augmentation. The generative formulation naturally supports controllable test-time sampling via reward guidance and anchored sampling, producing high-quality proposals. FlowR2A achieves state-of-the-art results on the NAVSIM v1 and v2 benchmarks, with multimodal proposals of substantially higher quality than prior methods. 1 Introduction End-to-end autonomous driving (E2E-AD) has emerged as a promising paradigm that maps raw sensor input directly to planning output through a differentiable model 20 ; 22 ; 7 . A typical E2E-AD model consists of a perception encoder that extracts scene observations from sensor input and a plan decoder that produces planning actions on top of these observations. Early efforts predict a single action that imitates the human trajectory 20 ; 22 ; 7 ; 6 ; 47 ; 30 ; 34 . Given the inherent uncertainty and multimodality of driving behavior, recent research has shifted toward multimodal planning 5 ; 32 ; 35 ; 17 , where the final action is selected from multiple proposals. Existing multimodal planners fall into two paradigms: scoring-based and anchor-based methods, illustrated in Fig. 1 . Scoring-based methods 5 ; 32 ; 28 ; 49 ; 33 use a large, fixed action vocabulary as candidates and train a scorer to evaluate each candidate with simulation-based reward labels, selecting the best one as output. The key insight of this paradigm is that dense reward supervision over the entire action vocabulary provides a rich, comprehensive signal about the relationship between actions and their outcomes. However, scoring-based methods are fundamentally discriminative: they learn p(r|a) to rank actions, but cannot transfer this knowledge to generate new proposals. Their output is therefore constrained to the fixed vocabulary, limiting adaptability to dynamic real-world scenarios. Anchor-based methods 17 ; 35 ; 52 ; 24 address this rigidity by decoding proposals dynamically from a set of action anchors and applying a winner-takes-all loss that supervises only the proposal closest to the single ground-truth (GT) trajectory. While this enables scene-adaptive proposals, the sparse GT supervision introduces two fundamental limitations. First, many anchors receive no training signal per scene, resulting in low-quality or degenerate proposals as shown in our experiments (Tab. 3 , Fig. 6 ). Second, targeting the single GT trajectory inherits well-known imitation learning pathologies 9 ; 10 , including shortcut learning from ego status and unawareness of action consequences. The two paradigms reveal a common tension: dense supervision and generative proposal modeling have so far been mutually exclusive. Scoring-based methods enjoy dense action-reward supervision but are bounded by their discriminative nature. Anchor-based methods can generate proposals, yet are supervised by a single GT trajectory per scene. This raises a natural question: can we combine dense reward supervision with generative proposal modeling in a single framework? In this work, we propose FlowR2A, a multimodal planning framework that learns the reward-conditioned action distribution p(a|r) from dense trajectory-reward pairs, resolving the above tension with a simple reframing. A dense action vocabulary can be paired with simulation-based rewards characterizing safety, progress, comfort, and rule compliance, yielding dense action-reward pairs that span the action space. While prior work 32 ; 49 treats these rewards as discriminative targets to predict, we instead treat them as conditions to learn the generative reward-to-action distribution. Under this formulation, each action-reward pair becomes a valid training sample, and the model is forced to internalize the correlation between an action and its outcomes. We construct fine-grained reward signals from rule-based simulation 10 and train a flow-matching-based 38 ; 36 ; 12 action decoder to recover clean trajectories from noise under reward conditioning. At inference, this generative formulation enables classifier-free guidance conditioned on high rewards and anchored sampling from any reference trajectory, yielding diverse, high-quality proposals. A practical challenge arises in conditioning on high rewards. High-reward actions often sit close to the boundary of the feasible region, requiring the generative decoder to approach it without crossing into the infeasible side. We address this through two designs. First, we replace key safety and compliance rewards with per-timestep counterparts, which provide a more general and finer-grained signal that sharpens the hard constraints. Second, we corrupt the continuous reward values with minor Gaussian noise during training, which acts as label smoothing and prevents the decoder from over-relying on them. Together, these designs effectively balance the conflicting objectives. Figure 1 : Comparison of multimodal planning paradigms. (a) Scoring-based methods select from a large fixed trajectory vocabulary. (b) Anchor-based methods decode multiple proposals from action anchors and sparsely supervise them with GT. (c) Our method FlowR2A learns the reward-conditioned action distribution p(a|r) from dense trajectories, where a is the action and r is the reward. Red lines indicate training supervision. We evaluate FlowR2A on both NAVSIM v1 and v2 benchmarks 10 ; 3 , achieving state-of-the-art performance. More importantly, our model produces multimodal proposals of substantially higher quality than prior methods, validating the benefit of modeling the entire conditional action distribution. Our contributions are: • A new paradigm for multimodal driving planning that learns the reward-to-action distribution p(a|r) , unifying the dense supervision of scoring-based methods with the generation ability of anchor-based methods, and naturally supporting controllable test-time sampling. • A reward construction recipe that effectively balances conflicting objectives with fine-grained per-timestep reward conditioning and reward noise augmentation. • State-of-the-art results on the NAVSIM benchmark with higher proposal quality than previous methods, validating the advantage of learning the full conditional action distribution. 2 Preliminaries Flow matching 36 ; 1 ; 38 provides a general perspective on generative modeling by defining a probability path between the noise distribution and the data distribution. It defines a forward process that linearly combines the data sample {\bm{x}}\sim p_{\mathrm{data}}({\bm{x}}) and the noise \bm{\epsilon}\sim\mathcal{N}(0,{\bm{I}}) into a noisy sample {\bm{z}}_{t}=a_{t}{\bm{x}}+b_{t}\bm{\epsilon} , where a_{t},b_{t} are noise schedules at time t\in[0,1] . Following rectified flow 38 , we use a straight path with a linear schedule, {\bm{z}}_{t}=t{\bm{x}}+(1-t)\bm{\epsilon}, (1) so {\bm{z}}_{0}=\bm{\epsilon} is pure noise and {\bm{z}}_{1}={\bm{x}} is the clean sample. The flow velocity is the time derivative of {\bm{z}}_{t} , {\bm{v}}=\frac{d{\bm{z}}_{t}}{dt}={\bm{x}}-\bm{\epsilon}. (2) Flow-based methods 12 ; 36 train a model {\bm{v}}_{\theta}({\bm{z}}_{t},t) to match {\bm{v}} via the velocity-matching loss \mathcal{L}=\mathbb{E}_{t,{\bm{x}},\bm{\epsilon}}\,\|{\bm{v}}_{\theta}({\bm{z}}_{t},t)-{\bm{v}}\|^{2}. (3) At inference, we draw clean samples by solving the ODE d{\bm{z}}_{t}={\bm{v}}_{\theta}({\bm{z}}_{t},t)\,dt from {\bm{z}}_{0}\sim\mathcal{N}(0,{\bm{I}}) at t=0 to {\bm{z}}_{1} at t=1 . We use a 20-step Euler solver 12 ; 13 in this work. 3 Method FlowR2A

Read the complete text