FROST-Drive: Scalable and Efficient End-to-End Driving with a Frozen Vision Encoder Zeyu Dong, Yimin Zhu, Yu Wu, Yu Sun Abstract End-to-end (E2E) models in autonomous driving aim to directly map sensor inputs to control commands, but their ability to generalize to novel and complex scenarios remains a key challenge. The common practice of fully fine-tuning the vision encoder on driving datasets potentially limits its generalization by causing the model to specialize too heavily in the training data. This work challenges the necessity of this training paradigm. We propose FROST-Drive, a novel E2E architecture designed to preserve and leverage the powerful generalization capabilities of a pretrained vision encoder from a Vision-Language Model (VLM). By keeping the encoder’s weights frozen, our approach directly transfers the rich, generalized world knowledge from the VLM to the driving task. Our model architecture combines this frozen encoder with a transformer-based adapter for multimodal fusion and a GRU-based decoder for smooth waypoint generation. Furthermore, we introduce a custom loss function designed to directly optimize for Rater Feedback Score (RFS), a metric that prioritizes robust trajectory planning. We conduct extensive experiments on Waymo Open E2E Dataset, a large-scale datasets deliberately curated to capture the long-tail scenarios, demonstrating that our frozen-encoder approach significantly outperforms models that employ full fine-tuning. Our results provide substantial evidence that preserving the broad knowledge of a capable VLM is a more effective strategy for achieving robust, generalizable driving performance than intensive domain-specific adaptation. This offers a new pathway for developing vision-based models that can better handle the complexities of real-world application domains. 1 Introduction End-to-end approaches to autonomous driving have emerged as a promising research direction, aiming to directly map raw sensor inputs to vehicle control commands. This paradigm seeks to overcome the limitations of traditional modular pipelines, which can suffer from compounding errors and the need for extensive manual tuning of individual components [ 1 , 3 ] . The dominant paradigm in this domain has converged on a modularized E2E design, which typically consists of a powerful vision encoder, a planner, and a decoder that generates trajectory waypoints [ 26 , 27 , 12 ] . In most contemporary studies, this entire architecture, including the vision encoder, is trained jointly from end to end. Figure 1 : Comparison of the model performance for different vision encoder approaches on Waymo E2E Dataset; ViT (ImageNet): use a frozen ViT pre-trained with ImageNet dataset; ViT (Finetune): fine-tune the E2E model end-to-end with Waymo Dataset; VLM: use a frozen ViT from a 72B VLM. Figure 2 : Comparison of the model performance using different sizes of visual embeddings on Waymo E2E Dataset; X axis: end-to-end model using a ViT with different embedding sizes. Fine-tuning or retraining the vision encoder has long been considered necessary for two main reasons. 1) There is a significant domain gap because models pre-trained on general datasets are not optimized for autonomous driving. Fine-tuning is thus required to adapt the feature extractor to recognize these domain-specific features; as shown in Figure 1 , a fine-tuned Vision Transformer (ViT) improves model performance compared to a ViT pre-trained only on ImageNet. 2) The representational capacity of standard pre-trained models is often considered insufficient for the complexity of the driving task. As illustrated in Figure 2 , performance scales with the embedding size, as larger embeddings carry richer information. Consequently, to utilize more powerful models with larger embeddings, retraining has been a prepared approach. However, the practice of fully fine-tuning the vision encoder presents a fundamental trade-off that can harm a model’s ability to generalize. While the process is computationally intensive, a more critical issue is that training end-to-end on a specific driving dataset limits the model’s exposure to the long-tailed distribution of real-world events [ 11 , 14 ] . By intensely specializing in the most common scenarios within the training data, the model’s broader, pre-trained knowledge can be compromised, weakening its ability to handle rare but critical situations. This creates a model that lacks the robust, generalizable understanding required for safe real-world deployment. This dilemma is a recurring challenge when adapting large-scale vision models to new application domains. To resolve this, we propose a new approach that leverages the rich world knowledge embedded within a large-scale foundation model. Instead of training a vision encoder from scratch or extensively fine-tuning it, we adopt a powerful, pre-trained vision encoder and keep its weights frozen. The rationale is that vision encoders co-trained with language on vast datasets develop a richer, more contextual understanding of the world. Additionally, these encoders are designed to generate high-dimensional feature embeddings capable of carrying the large volume of information required for complex driving scenarios. As a result, they are ideal for downstream tasks that require intricate reasoning. We argue that these two advantages—a better understanding of the world and a high-capacity feature representation—make it unnecessary to fine-tune the vision model. We validate our hypothesis through extensive experiments on the Waymo Open E2E Dataset, a dataset corated to address long-tail scenarios in different environments. Our results demonstrate that a model utilizing a frozen vision encoder from a large foundation model significantly outperforms conventional approaches. Specifically, it surpasses the performance of both a fully fine-tuned model and models that use a frozen encoder pre-trained only on a general-purpose dataset. These findings suggest that retraining the vision component can be unnecessary for achieving state-of-the-art performance, provided that we transfers knowledge from a sufficiently powerful base model. The primary contributions of this work are as follows: 1. We designed the FROST-Drive architecture for autonomous driving, featuring a frozen vision encoder, a transformer adapter, and a GRU decoder, all optimized with a custom RFS loss function. This approach serves as a generalizable and efficient alternative to full end-to-end training, achieving a top-three rank in the Waymo End-to-End Driving Challenge. 2. We provide critical insight into a common paradigm in end-to-end autonomous driving, showing that retraining a vision model may be unnecessary when the vision encoder is sourced from a sufficiently powerful VLM. 3. We are the first to conduct a comprehensive set of experiments analyzing how vision encoder size and feature embedding dimensionality affect the performance of a frozen-encoder approach, revealing a clear scaling relationship that validates the importance of using high-dimensional representations for this complex task. 2 Related Work Model Design in E2E Autonomous Driving Unlike traditional modular approaches, end-to-end autonomous driving aims to generate driving actions directly from raw sensor data [ 3 ] . The field has flourished since NVIDIA introduced PilotNet [ 1 ] , which utilized a simple 5-layer CNN as its image encoder. To handle more challenging driving scenarios, a variety of more complex architectures have been proposed. These include using ViT [ 28 ] or ResNet [ 16 ] as the image backbone, using swin transformer to generate video token for further text generation and control signal prediction [ 15 ] , employing transformers for multi-sensor fusion [ 8 , 26 ] , incorporating the intermediate outputs and a holistic token mixer sub-network for effective feature adaptation [ 9 ] , adopting Bird’s-Eye View (BEV) image encoders [ 27 , 7 , 12 ] , and fine-tuning VLMs [ 25 ] . Over time, the architectural design for end-to-end systems has largely converged on a modularized pipeline, typically featuring a vision backbone for feature extraction, a planning module, and a waypoint decoder [ 2 , 12 ] . However, a common thread among most of these approaches is the requirement to fully fine-tune the entire model, including the large and computationally intensive vision module. This process demands extensive resources and large datasets, and it introduces a significant risk of overfitting to the training data. Use of VLM in Autonomous Driving The integration of VLMs has recently become a vibrant research frontier, leveraging their advanced reasoning and semantic understanding to enhance autonomous systems. Models like DriveGPT4 [ 33 ] , RAG-Driver [ 34 ] , and DriveCoT [ 31 ] to generate natural language CoT reasoning for driving actions. Beyond reasoning in natural language, researchers are increasingly using VLMs for motion planning and control. Some approaches employ hierarchical agents to enable real-world applications: LM-Nav [ 24 ] and ViNG [ 23 ] use a VLM as a visual guide for navigation. There are also dual-system designs where a high-level VLM makes strategic decisions that guide a low-level controller [ 36 , 30 , 10 ] . Others aim for a fully unified model; for instance, GPT-Driver [ 21 ] and EMMA [ 13 ] represent a significant push towards all-in-one models that can process multimodal inputs and directly output driving commands. A common theme across these E2E approaches, however, is the reliance on extensive fine-tuning to adapt the general-purpose VLM to the specific demands of E2E vehicle control. 3 Methodolgy Figure 3 : Model Architecture The primary objective of this work is to develop an end-to-end autonomous driving system that can safely and robustly navigate complex real-world environments. We formulate this as a trajectory planning task. Given a set of multi-view camera images \mathcal{I}=\{I_{1},\ldots,I_{N_{c}}\} from N_{c} cameras at the current time T , a high-level, discrete driving intent c , a history of past states S_{past}=(s_{T-k},...,s_{T-1}) and current state s_{T} , where each s_{i} may contains several features like the position, speed, origination, etc. Our goal is to predict a sequence of local future waypoints \hat{W}=(\hat{w}_{1},...,\hat{w}_{H}) over a prediction horizon H as a sequence of 2D-coordinates relative to the current location of the car. The core task is to learn a mapping function that generates a trajectory that is not only accurate but also safe and comfortable for passengers. To achieve this, we propose a model designed for both high performance and training efficiency. As illustrated in Figure 3 , our architecture is composed of four primary components: a frozen, pretrained vision encoder from a VLM for feature extraction; a transformer-based adapter for fusing multi-camera input; a transformer encoder for fusing other multi-modal data; and a Gated Recurrent Unit (GRU) decoder for predicting the future waypoints. Another key aspect of our approach is the use of a novel loss function specifically designed to optimize for the Rater Feedback Score (RFS) [ 19 ] , a critical metric for ensuring safe and reliable trajectory planning. 3.1 Introduction to Rater Feedback Score In order to calculate the RFS, we define the reference trajectories, provided by dataset, as preferred rater specified trajectories, with a score \bar{r} for each. Suppose the preferred rater specified trajectories are W=(w_{1},w_{2},...,w_{H}) . And the predicted waypoints are \hat{W}=(\hat{w}_{1},\hat{w}_{2},...,\hat{w}_{H}) as defined in previous paragraph. With given preferred rater specified trajectories, we therefore define trust regions . A trust region is defined for the region within a given lateral and longitudinal threshold of the rater-specified trajectory at a given time t (t=3 s, 5 s are used in this work). The size of the trust region depends on the time step of the waypoint ( time-based thresholds ) and the velocity at the corresponding time ( speed-based scaling ) in both lateral and longitudinal directions and we finally derive the trust region by following steps: Time-based thresholds. The raw lateral/longitudinal thresholds \tilde{\tau}_{\mathrm{lat}},\tilde{\tau}_{\mathrm{lng}} (in meters) are defined in Table 1 . Time t \tilde{\tau}_{\mathrm{lat,t}} \tilde{\tau}_{\mathrm{lng,t}} 3 1.0 4.0 5 1.8 7.2 Table 1 : Trust region tolerance values over time Speed-based scaling. The raw thresholds are scaled by the velocity(m/s) of the preferred rater specified trajectory: \mathrm{scale}(v)=\begin{cases}0.5,&v<1.4,\\[6.0pt] 0.5+0.5\times\frac{v-1.4}{11-1.4},&1.4\leq v<11,\\[10.0pt] 1,&v\geq 11.\end{cases} The choice of speed for different scaling comes from the suggestions from [ 29 ] . The speed 1.4m/s is around 5kmh (or 3 mph), in which pedestrian crash fatality is relatively low, while 11 m/s is around 40 kmh (or 25mph), in which pedestrian crash fatality reachs around 10%. So we set up three intervals for the speed of safety, injury, fatality, accordingly. Final thresholds. Once we have the raw thresholds and the scale factors, the final thresholds are computed by \tau_{\mathrm{lat}}(t,v)=\mathrm{scale}(v)\,\tilde{\tau}_{\mathrm{lat,t}} and \tau_{\mathrm{lng}}(t,v)=\mathrm{scale}(v)\,\tilde{\tau}_{\mathrm{lng,t}}. If the predicted waypoint is inside the trust region, RFS is assigned to be full score. However, if the predicted waypoint is outside of the trust region, we calculate an exponentially decreasing score for this waypoint. The overall formula is: \displaystyle\text{RFS}(w_{t},\hat{w_{t}})=\begin{cases}\bar{r}&,\quad\text{when}\quad\Delta\leq 1,\\ \displaystyle\bar{r}\times 0.1^{\Delta-1}&,\quad\text{otherwise}.\end{cases} where \Delta\doteq\max\left\{\dfrac{\Delta_{\mathrm{lat,t}}}{\tau_{\mathrm{lat,r}}},\dfrac{\Delta_{\mathrm{lng,t}}}{\tau_{\mathrm{lng,t}}}\right\} is the maximum distance error among lateral or longitudinal directions, \Delta_{\mathrm{lat,t}} and \Delta_{\mathrm{lng,t}} are lateral and longitudinal distance errors between the predicted waypoint \hat{w}_{t} and corresponding waypoint w_{t} in preferred rater specified trajecotory at time t , and \bar{r} is the full score of preferred rater specified trajectory. Besides RFS, Average Distance Error (ADE) is used secondary metric in this work. For each predicted trajectory, ADE is defined as the Euclidean distance between the prediction and the highest-scored rater trajectory at each time step and calculate average. Compared to conventional metrics like ADE, RFS is customized for more realistic autonomous driving scenarios. It is speed-aware, allowing for larger tolerable errors at higher speeds to reflect real-world physics. It also features deviation tolerance, defining a ”trust region” that rewards any safe trajectory, not just perfect imitation of a single path. Finally, RFS has a strong emphasis on safety, as the score decays exponentially for any waypoint outside this region, heavily penalizing potentially dangerous deviations. 3.2 Model Architecture As illustrated in Figure 3 , the architecture is composed of four primary components: a frozen, pretrained vision encoder from a VLM for feature extraction; a transformer-based adapter for fusing multi-camera input; a transformer encoder for fusing other multi-modal data; and a GRU decoder for predicting the future waypoints. Pretrained Vision Encoder The foundation of our model is a powerful, frozen vision encoder, denoted as f_{ViT} , which is sourced from InternVL3 [ 4 , 32 , 5 , 6 ] , a state-of-the-art VLM. InternVL3 excels in multimodal perception and reasoning, making its vision encoder uniquely suited for complex scene understanding. The vision model uses a pixel-unshuffle technique to support high-resolution image data while producing a high-dimensional embedding. For our task, each camera view I_{j}\in\mathcal{I} in the multi-camera setup is resized to 448x448 pixels. Each image I_{j} is then processed independently by the encoder to extract a corresponding high-dimensional image feature, or visual embedding, E_{img,j}=f_{ViT}(I_{j}) . The resulting embedding has a shape of L_{img}\times d_{img} , where L_{img} is the number of visual tokens and d_{img} is the embedding dimension. By keeping the encoder weights frozen, we leverage the rich world knowledge learned during its extensive pretraining, avoiding the computational cost and overfitting risks associated with fine-tuning. Transformer Adapter Inspired by the BLIP-2 architecture [ 17 ] , we employ a transformer-based adapter that uses a set of learnable query tokens to efficiently fuse information from the multi-view cameras and condense the visual information. First, the high-dimensional visual embedding E_{img,j} from each camera view I_{j} is passed through a linear projection layer, f_{proj} , to reduce its feature dimension from d_{img} to a smaller dimension d_{model} . This step, producing E^{\prime}_{img,j}=f_{proj}(E_{img,j}) , significantly reduces the number of parameters in the trainable parts of our model. Next, the projected visual tokens from all N_{c} camera views are concatenated along the token dimension to form a single feature sequence: E^{\prime}_{img}=[E^{\prime}_{img,1};E^{\prime}_{img,2};\ldots;E^{\prime}_{img,N_{c}}] . To compress this long sequence, we utilize a set of L_{\mathcal{I}} learnable query tokens, Q_{\mathcal{I}}\in\mathbb{R}^{L_{\mathcal{I}}\times d_{model}} . We perform cross-attention where Q_{\mathcal{I}} acts as the query and E^{\prime}_{img} serves as both the key and value. This allows the model to distill the most salient information from the entire visual context into a compact representation E_{\mathcal{I}}=\text{CrossAttn}(Q_{\mathcal{I}},E^{\prime}_{img},E^{\prime}_{img}). The resulting fused image embedding, E_{\mathcal{I}} , has a fixed shape of L_{\mathcal{I}}\times d_{model} , where L_{\mathcal{I}}