Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Driving
Luke Rowe, Rodrigue de Schaetzen, Roger Girgis, Christopher Pal, Liam Paull · 2025
Read Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Driving by Luke Rowe, Rodrigue de Schaetzen, Roger Girgis, Christopher Pal, Liam Paull (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11234.txt.
Introduction
Luke Rowe and four co-authors present Poutine, a self-driving system built on a vision-language model, in a 2025 study. Odd road scenes defeat many driving stacks, so the team treat them as reasoning tasks for a general model. Poutine starts from a stock three-billion-parameter model with no added parts. It first learns by predicting the next token across image, text, and path data, drawn from both common and rare scenes.
A light round of preference-based reinforcement tuning then sharpens the driver. On a benchmark of hard scenes from a major driving data set, the final model took first place in a 2025 contest by a wide margin. The authors say the win shows that fancy add-ons are not needed for strong driving. Careful pre-training plus a small tuning set, they argue, carries the day.
Catalog record
| CREATOR | Luke Rowe, Rodrigue de Schaetzen, Roger Girgis, Christopher Pal, Liam Paull |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-11234
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Poutine: Vision-Language-Trajectory Pre-Training and Reinforcement Learning Post-Training Enable Robust End-to-End Autonomous Driving Luke Rowe, Rodrigue de Schaetzen, Roger Girgis, Christopher Pal, Liam Paull Abstract We present Poutine, a 3B-parameter vision-language model (VLM) tailored for end-to-end autonomous driving in long-tail driving scenarios. Poutine is trained in two stages. To obtain strong base driving capabilities, we train Poutine-Base in a self-supervised vision–language–trajectory (VLT) next-token prediction fashion on 83 hours of CoVLA nominal driving and 11 hours of Waymo long-tail driving. Accompanying language annotations are auto-generated with a 72B-parameter VLM. Poutine is obtained by fine-tuning Poutine-Base with Group Relative Policy Optimization (GRPO) using less than 500 preference-labeled frames from the Waymo validation set. We show that both VLT pretraining and RL fine-tuning are critical to attain strong driving performance in the long-tail. Poutine-Base achieves a rater-feedback score (RFS) of 8.12 on the validation set, nearly matching Waymo’s expert ground-truth RFS. The final Poutine model achieves an RFS of 7.99 on the official Waymo test set, placing 1st 1 1 1 Our submission topped the public leaderboard but, under the competition’s Quebec-residency exclusion, was not prize-eligible; the organizers instead granted us a special mention. in the 2025 Waymo Vision-Based End-to-End Driving Challenge by a significant margin. These results highlight the promise of scalable VLT pre-training and lightweight RL fine-tuning to enable robust and generalizable autonomy. 1 Introduction Vision–language models (VLMs) have emerged as a powerful means of coupling visual perception with the world knowledge and common sense reasoning acquired from internet-scale pre-training ( Bai et al. 2023 ; OpenAI 2023 ; Chen et al. 2023 ; Llama Team 2024 ; Google 2024 ) . For autonomous vehicles, such multimodal reasoning is most valuable in long-tail situations—rare but safety-critical events that dominate operational risk and have limited coverage in conventional driving corpora. Nevertheless, the empirical study of VLMs for driving has so far been restricted largely to nominal driving benchmarks such as nuScenes ( Caesar et al. 2020 ) , where high-level semantic reasoning is seldom required and the benefits of language grounding remain unclear ( Hegde et al. 2025 ; Jiang et al. 2024 ; Pan et al. 2024 ; Sima et al. 2024 ; Nie et al. 2024 ; Bai et al. 2024 ; Tian et al. 2024b ; Tian et al. 2024a ; Xing et al. 2024 ; Huang et al. 2024 ; Wang et al. 2024 ; Zhou et al. 2025 ; Mao et al. 2023 ; Hwang et al. 2024 ; Xu et al. 2024 ) . The Waymo Vision-Based End-to-End Driving (WOD-E2E) dataset, along with its accompanying 2025 challenge, presents a new opportunity to evaluate VLMs in more challenging, curated long-tail scenarios. These characteristics make WOD-E2E a better testbed to investigate whether the knowledge embedded in VLMs can translate into safer and more reliable driving policies. Figure 1 : At the end of the submission deadline, Poutine ranked first on the 2025 Waymo Vision-Based E2E Driving Challenge by a considerable margin. Its pre-reinforcement learning variant, Poutine-Base, also outperformed all other submitted entries. To address this question, we introduce Poutine , a 3B-parameter VLM for E2E driving, trained via a simple two-stage pipeline. In Stage 1, Poutine-Base learns base driving capabilities by training via next-token prediction over vision, language, and future-trajectory (VLT) tokens. The training corpus comprises 83 hours of nominal Japanese driving from the public CoVLA dataset ( Arai et al. 2025 ) and 11 hours of long-tail driving from WOD-E2E. All language annotations are generated automatically by a 72B-parameter VLM, removing the need for manual labeling and yielding a fully self-supervised pre-training procedure that is straightforward to scale. In Stage 2, we refine Poutine-Base with Group Relative Policy Optimization (GRPO) using less than 500 human preference-labeled frames from the WOD-E2E validation set. Despite the modest supervision budget, this lightweight reinforcement-learning (RL) step materially improves policy performance in the long-tail. The trained Poutine-Base model achieves a rater-feedback-score (RFS) of 8.12 on the WOD-E2E validation set, nearly matching the 8.13 RFS of Waymo’s expert ground-truth trajectories. Moreover, a variant trained solely on the CoVLA Japanese driving dataset generalizes zero-shot to U.S. driving data, achieving an RFS of 7.74 on the WOD-E2E validation set despite never encountering Waymo data during pre-training. This result highlights the potential of collecting diverse driving data from various geographical regions to train a unified driving policy via large-scale next-token prediction VLT pre-training. After GRPO fine-tuning, the final Poutine model scores 7.99 RFS on the official WOD-E2E test set, achieving first place in the 2025 Waymo Vision-Based End-to-End Driving Challenge by a large margin (see Figure 1 ). Collectively, our results indicate that scalable VLT pre-training, coupled with lightweight preference-based RL, constitutes a practical recipe for robust and generalizable autonomy in challenging long-tail driving scenarios. 2 Poutine 2.1 Language Annotation Dataset Collection Language annotations were automatically generated on the WOD-E2E and CoVLA driving datasets using a pretrained Qwen2.5-VL 72B Instruct model 2 2 2 The original CoVLA captions lacked semantic diversity which prompted us to generate new captions. . This procedure required no manual labelling or post-hoc verification, thereby eliminating human effort and improving the scalability of our approach. Each prompt supplied the VLM with three camera frames over the last second, the high-level driving command ( intent ) at the current frame, and the 4-second past trajectory. In contrast to most driving datasets with language and action data, we also conditioned on the ground-truth 5-second future trajectory of the ego vehicle. The resulting captions therefore explain why the vehicle executes the given future trajectory rather than predict where the ego should go based on the past context. We found that this significantly improves the consistency between the language annotations and ground-truth future trajectories. We divided the language annotation task into three components: 1) identifying the relevant critical objects from a predefined list, 2) composing a short description explaining why the expert trajectory was executed, and 3) identifying the meta behavior from a predefined list of possible speed and command actions. Preliminary testing showed that the annotation quality dramatically improves when the critical object detection task is structured as yes or no questions compared to more open-ended descriptions of the task. The system prompt is shown in Figure 3 . We generated 240.5 K high-quality captions for the WOD-E2E (training set) and CoVLA datasets (sampled at 1 Hz), using 12 A100 GPUs over 24 h. To accelerate VLM inference, front-view images from the current frame were resized to 532 × 476 px, while all other images were limited to at most 256 × 256 px. Figure 2 shows three examples of our automatically generated annotations. Figure 2 : Generated annotations on WOD-E2E data. The red dots depict the 5-second future trajectory. Only two views from the current frame and the explanation of the annotation are shown. Bold text highlights the objects and meta behavior selected by the model. ⬇ You are an expert labeller of driving scenarios . Input : - 3 frames of multi - view images collected from the ego - vehicle over the last 1 second - Current high - level intent ( string ) - 4- second past trajectory (16 steps at 4 Hz ) - Expert 5- second future trajectory (20 steps