Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving
Linhan Wang, Zichong Yang, Chen Bai, Guoxiang Zhang, Xiaotong Liu, Xiaoyin Zheng, Xiao-Xiao Long, Chang-Tien Lu, Cheng Lu · 2026
Read Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving by Linhan Wang, Zichong Yang, Chen Bai, Guoxiang Zhang, Xiaotong Liu, Xiaoyin Zheng, Xiao-Xiao Long, Chang-Tien Lu, Cheng Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-22032.txt.
Introduction
Linhan Wang and co-authors teach self-driving systems from video. End-to-end driving models increasingly pretrain on video without labels, hoping to learn transferable planning skills. So far that pretraining has brought only limited gains. Driving is also deeply ambiguous: one scene usually comes with just a single human trajectory as the example.
The team's 2026 preprint, Drive-JEPA, combines a video prediction architecture with multimodal trajectory distillation. Distilling several plausible trajectories gives the model a richer picture of what good driving could look like in the same scene. Better pretraining matters because labeled driving data are expensive, while unlabeled dashcam video exists in enormous quantities.
Catalog record
| CREATOR | Linhan Wang, Zichong Yang, Chen Bai, Guoxiang Zhang, Xiaotong Liu, Xiaoyin Zheng, Xiao-Xiao Long, Chang-Tien Lu, Cheng Lu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-22032
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving Linhan Wang, Zichong Yang, Chen Bai, Guoxiang Zhang, Xiaotong Liu, Xiaoyin Zheng, Xiao-Xiao Long, Chang-Tien Lu, Cheng Lu Abstract End-to-end autonomous driving increasingly leverages self-supervised video pretraining to learn transferable planning representations. However, pretraining video world models for scene understanding has so far brought only limited improvements. This limitation is compounded by the inherent ambiguity of driving: each scene typically provides only a single human trajectory, making it difficult to learn multimodal behaviors. In this work, we propose Drive-JEPA, a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving. First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning. Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior. When evaluated on NAVSIM, the V-JEPA representation combined with a simple transformer-based decoder outperforms prior methods by 3 PDMS in the perception-free setting. The complete Drive-JEPA framework achieves 93.3 PDMS on v1 and 87.8 EPDMS on v2, setting a new state-of-the-art. The code is available at https://github.com/linhanwang/Drive-JEPA . † † affiliationnotice: * Work done during an internship at XPENG Motors. 1 Introduction End-to-end autonomous driving ( 34 ; 11 ; 22 ) has emerged as a promising paradigm that directly maps raw sensor observations to driving actions using a unified neural model. By eliminating hand-designed intermediate representations used in traditional modular pipelines, end-to-end approaches aim to reduce information loss and improve scalability by learning directly from large collections of human driving data. Figure 1 : Comparison between end-to-end planners on both perception-free and perception-based settings. Recently, end-to-end autonomous driving increasingly seeks to leverage self-supervised video pretraining to learn transferable representations for planning. However, pretraining video world models for scene understanding has so far brought only limited improvements. Existing approaches in this direction largely fall into two categories. First, video-generative methods, such as VaVAM ( 5 ) and Epona ( 45 ) , learn representations by reconstructing or generating videos and then transfer them to planning, but this pixel-level objective incurs heavy computation and may over-emphasize visual details that are irrelevant to decision making. Second, to reduce cost, latent world models predict compact feature dynamics (e.g., LAW ( 28 ) predicts feature T{+}1 from feature T , and World4Drive ( 46 ) further introduces pretrained foundation models to enrich latent targets. However, these latent approaches are typically used as auxiliary objectives and have not demonstrated clear benefits from scaling up pretraining. Orthogonally, end-to-end driving faces a supervision bottleneck: each scene typically provides only a single human trajectory, despite inherently multimodal futures. Prior works address this by generating multimodal trajectories through either discrete or continuous formulations. Discrete approaches such as VAD v2 ( 10 ) and Hydra-MDP ( 29 ) cluster trajectories into a fixed vocabulary and predict scores reflecting safety and comfort; however, their expressiveness is fundamentally limited by the coverage and quality of anchor trajectories, leading to poor generalization in out-of-vocabulary scenarios ( 31 ) . Alternatively, diffusion-based methods, including DiffusionDrive ( 31 ) and GoalFlow ( 41 ) , model multimodal trajectory distributions via iterative sampling, which has shown strong generative capability. Nevertheless, these approaches remain constrained by supervision from single human trajectories per scene, inherently limiting the diversity of learned behaviors. In this work, we propose Drive-JEPA , an end-to-end autonomous driving framework that addresses the above two bottlenecks in a unified way. First, we adapt V-JEPA ( 2 ; 3 ) to the driving domain to learn planning-aligned predictive representations from large-scale raw videos, improving transfer beyond prior world-model pretraining. Second, we introduce multimodal trajectory distillation that distills knowledge from simulators into a proposal-centric planner, providing diverse supervision beyond single human trajectories and enabling safer multimodal decision making. Specifically, our framework consists of three components: Driving Video Pretraining, Multimodal Trajectory Distillation, and Momentum-aware Trajectory Selection. In the first module, we curate a large-scale driving video dataset and pretrain a ViT-based vision encoder using V-JEPA ( 2 ; 3 ) , which learns predictive representation by predicting future latent with effective mode collapse prevention. In the second module, the waypoint-anchored proposals generation leverages deformable attention ( 40 ; 18 ) to aggregate BEV features ( 30 ) at trajectory waypoints and refine proposals iteratively. To increase diversity, proposals are supervised using both human trajectories and simulator-generated multimodal trajectories that satisfy safety and comfort constraints, enabling effective knowledge distillation from the simulator. Finally, the selection module assigns scores to all candidates by predicting collision risk, traffic-rule compliance, and comfort, and further incorporates a momentum-aware penalty to reduce frame-to-frame trajectory distortion. We validate Drive-JEPA on NAVSIM v1 ( 12 ) , NAVSIM v2 ( 8 ) , and Bench2Drive ( 23 ) . Drive-JEPA achieves 93.3 PDMS on NAVSIM v1 and 87.8 EPDMS on NAVSIM v2, setting a new state of the art. Notably, with only a single front-view camera and a lightweight transformer planner, our V-JEPA-pretrained model outperforms prior work by 3 PDMS in the perception-free setting, highlighting the effectiveness of V-JEPA pretraining for planning. On Bench2Drive, the Multimodal Trajectory Distillation consistently improves driving quality, demonstrating the benefit of diverse supervision for generating safe, multimodal trajectories. Our contributions can be summarized as follows: • We introduce V-JEPA pretraining to end-to-end autonomous driving, boosting performance in both perception-based and perception-free settings. • We propose a novel multimodal trajectories supervision to distill simulator knowledge to a proposal-centric framework, generating diverse multimodal trajectories. • We design a momentum-aware trajectory selection module, enhancing driving comfort. • Our method achieves a new state of the art on NAVSIM v1 and NAVSIM v2. In addition, our method achieves strong performance on NAVSIM even without relying on perception annotations. 2 Related Work 2.1 End-to-end autonomous driving Early works such as ALVINN ( 34 ) and PilotNet ( 6 ) leverage large-scale human driving data to learn policies that map sensor observations directly to control actions. However, these models often lack interpretability and can degrade due to issues such as causal confusion. To mitigate this, recent studies incorporate intermediate representations and auxiliary supervision to improve robustness. Transfuser ( 11 ) fuses LiDAR and camera features in the BEV space and strengthens BEV features with BEV segmentation and 3D detection supervision. Going further, UniAD ( 22 ) unifies the full stack of driving tasks—including tracking, mapping, and motion prediction—within a single framework jointly optimized with planning. VAD ( 25 ) explores compact vectorized scene representations for efficiency, while SparseDrive ( 37 ) proposes a query-centric sparse structure as a BEV-free alternative. DriveTransformer ( 24 ) further improves efficiency by using a small set of learned queries to aggregate multi-view image features. Moreover, DiffusionDrive ( 31 ) and GoalFlow ( 41 ) investigate diffusion-based end-to-end trajectory generation. Despite these advances, end-to-end driving remains challenging due to two fundamental requirements: capturing the spatiotemporal structure of complex scenes and modeling the inherently multimodal nature of driving behaviors. 2.2 World Models for End-to-end driving Video world models in autonomous driving predict how scenes evolve under ego actions. Recent progress in controllable video generation ( 17 ; 21 ) has