SteerVLA: Steering Vision-Language-Action Models in Long-Tail Driving Scenarios Tian Gao, Celine Tan, Catherine Glossop, Timothy Gao, Jiankai Sun, Kyle Stachowicz, Shirley Wu, Oier Mees, Dorsa Sadigh, Sergey Levine, Chelsea Finn Abstract A fundamental challenge in autonomous driving is the integration of high-level, semantic reasoning for long-tail events with low-level, reactive control for robust driving. While large vision-language models (VLMs) trained on web-scale data offer powerful common-sense reasoning, they lack the grounded experience necessary for safe vehicle control. We posit that an effective autonomous agent should leverage the world knowledge of VLMs to guide a steerable driving policy toward robust control in driving scenarios. To this end, we propose SteerVLA, which leverages the reasoning capabilities of VLMs to produce fine-grained language instructions that steer a vision-language-action (VLA) driving policy. Key to our method is this rich language interface between the high-level VLM and low-level VLA, which allows the high-level policy to more effectively ground its reasoning in the control outputs of the low-level policy. To provide fine-grained language supervision aligned with vehicle control, we leverage a VLM to augment existing driving data with detailed language annotations, which we find to be essential for effective reasoning and steerability. We evaluate SteerVLA on a challenging closed-loop benchmark, where it outperforms state-of-the-art methods by 4.77 points in overall driving score and by 8.04 points on a long-tail subset. The project website is available at: https://steervla.github.io/ . Figure 1 : SteerVLA encountering long-tail scenarios . SteerVLA is able to quickly reason about and adapt to a traffic accident blocking the lane. It first slows down, then waits for a gap in the traffic, and merges when the lane is available. 1 Introduction Despite rapid progress in autonomous driving systems, long-tail scenarios remain particularly challenging due to their inherent scarcity in driving data and the complex reasoning they require. A truly autonomous vehicle must handle ambiguous traffic flow in construction zones, unpredictable pedestrian behavior, and blocked lanes due to accidents, as well as the compositions of these scenarios. For example, in Fig. 1 , the vehicle encounters an accident blocking the lane. It must first reason about the scenario and recognize that it cannot simply continue in the same lane. It then needs to decide the best course of action while considering the other vehicles on the road. Handling these long-tail scenarios effectively is fundamentally important to building safe and robust driving systems ( 45 ) . Vision–language–action (VLA) models, derived from vision–language models (VLMs) and adapted to driving control via imitation learning, leverage strong semantic priors to generate embodied actions ( 5 ; 26 ; 60 ) . However, long-tail driving scenarios often require reasoning over rare events, implicit social norms, and broader common-sense knowledge that can be very hard to infer reliably from immediate visual driving cues alone. Pretrained VLMs encode such semantic knowledge, but effectively applying it in driving depends on how these inferences are grounded in control. We focus on enabling semantic reasoning from VLMs to guide driving behavior in VLA policies, allowing effective use of pretrained knowledge in complex and long-tail driving scenarios. We present SteerVLA, a novel framework for VLA-based driving policies effective in both normal and long-tail scenarios. Our key insight is to steer VLA control using VLM reasoning through: 1) a high-level policy, fine-tuned from a pretrained VLM, that performs semantic and common-sense reasoning to analyze driving scenarios based on camera images, routing commands (e.g., “Turn left at the next intersection”) from navigation APIs, and historical vehicle states. This model outputs reasoning traces and meta-actions —driving instructions for ego-vehicle motion (e.g., “accelerate and make a wide left turn, cautiously monitoring the junction.”), and 2) a low-level policy, fine-tuned from a pretrained VLM to generate precise control actions conditioned on meta-actions. This design leverages the powerful reasoning capabilities of VLMs while producing fine-grained outputs for driving control in VLAs. The key challenges are enabling the high-level policy to reason over diverse driving scenarios and generate detailed commands, as well as training a low-level policy to reliably execute them in a steerable manner. For example, if an accident blocks the road and the car ahead enters the opposite driving lane, the high-level policy should reason that “The lane appears shared bidirectionally since the vehicle in front moved into the oncoming lane. Proceed cautiously” and output conservative acceleration from a stopped state. Given camera images and vehicle states, the low-level policy maps meta-actions to precise control actions such as speed control and steering angle. However, difficulty arises from the scarcity of natural language supervision in driving datasets, where language annotations are often missing or coarse, and rarely grounded in driving control. To provide high-quality language supervision, we design an automatic data generation pipeline that constructs language supervision explicitly grounded in control. Given driving scenes and their associated driving trajectories, the pipeline guides a VLM to generate or refine meta-actions that describe the underlying driving behaviors over a temporal window. The pipeline also augments images with mid-level representations (bounding boxes or trajectory projections) to help the VLM better reason about spatial relationships and align its understanding with driving control. Rather than relying on generic commands such as “decelerate due to the stop sign”, we enrich meta-actions with details derived from trajectories, including motion intensity and directional adjustments. This results in grounded descriptions such as “decelerate rapidly and cautiously make a slight right adjustment before stopping for a sign,” which better steer the low-level policy. Using this automatically generated supervision, we fine-tune the high-level policy to reason over complex scenes and produce well-grounded meta-actions. We train the low-level policy to follow these detailed meta-actions and imitate the safe driving behavior in our training data. The high-level policy can then control the low-level policy at the meta-action level, allowing for safe, intelligent responses in driving scenarios that require complex reasoning. This paper introduces a training and data generation framework that enables semantic reasoning to steer driving control, leading to strong performance in long-tail scenarios. We evaluate SteerVLA on the Bench2Drive ( 25 ) benchmark in the CARLA ( 12 ) simulator. To test long-tail performance specifically, we identify 11 long-tail scenarios in Bench2Drive, which we term Bench2Drive-LongTail. We find that SteerVLA outperforms state-of-the-art methods by 4.77 points in driving score on Bench2Drive overall, and notably by 8.04 points on Bench2Drive-LongTail, confirming that grounded reasoning improves generalization in long-tail scenarios. 2 Related Work We review related work in three areas: VLA–based driving models, methods that incorporate reasoning into driving models, and data labeling for driving data. Vision-language-action models in driving. Although autonomous driving has traditionally consisted of methods that use a stack of perception, prediction, and planning modules ( 19 ; 20 ; 41 ) , massive progress has been made with end-to-end imitation learning methods that directly map multi-modal inputs to driving commands ( 14 ; 33 ; 58 ; 18 ) . These methods generally excel in generic driving scenarios, but can struggle to generalize to long-tail scenarios, as these are not well-covered in driving data. Several works have gone beyond training end-to-end policies from scratch and leverage large language and vision-language models to leverage their pre-trained capabilities. Various works fine-tune pre-trained large language models on driving data ( 24 ; 54 ; 21 ; 1 ; 60 ; 15 ; 16 ; 61 ; 46 ; 38 ) . Some ( 8 ; 51 ) integrate multimodal inputs, such as images, by projecting them into token space, while others ( 30 ; 31 ; 35 ) adapt pre-trained VLMs as motion planners through text-based fine-tuning. Inspired by the success of pretrained vision-language models (VLMs), several works have introduced vision-language-action (VLA) models ( 5 ) , which consist of a VLM backbone fine-tuned to produce robot actions ( 26 ) . These models benefit from excellent cross-modal grounding between language and vision, enabling the transfer of internet-scale semantic knowledge from their pre-training data. However, a key challenge for these methods is retaining the strong capabilities learned during pre-training, which can be lost when transferring to the domain of action prediction, a task very different from those found in VLM pre-training ( 13 ) . While some of these methods fine-tune VLMs with an action head ( 22 ; 59 ; 45 ; 37 ) to mitigate this issue, we explicitly use a hierarchical model, allowing the high-level policy training to stay closer to VLM pre-training tasks. Moreover, we develop an auto-labeling pipeline for autonomous driving data, real or simulated, to provide dense language labels in the form of reasoning traces and detailed meta-action labels, to train the high-level and low-level policies. We find that while the hierarchical structure is essential for retention of the reasoning capabilities of the base VLM in the high-level policy, these dense labels are what allow for effective communication between the policies, which is key to SteerVLA’s performance in long-tail scenarios. Reasoning in Autonomous Driving. Recent works have sought to imbue VLAs with reasoning capabilities ( 55 ; 56 ; 32 ; 39 ; 2 ; 9 ; 42 ; 27 ; 53 ) to improve generalization and compositional task-following. In the driving domain, reasoning has been primarily used in the form of chain-of-thought steps ( 61 ; 35 ; 47 ; 28 ; 18 ; 51 ; 37 ) , casting reasoning as detecting other vehicles, describing the scene, performing question-answering tasks, or providing explainability or justification signals. While these methods improve reasoning or generalization, they remain largely descriptive. In contrast, we use our auto-labeling pipeline to generate both descriptive reasoning traces, specifically including the states of other vehicles and traffic sign information, and detailed prescriptive meta-action labels. Most similar to our work is SimLingo ( 37 ) , which also focuses on long-tail driving scenario capabilities and achieves state-of-the-art performance on the Bench2Drive benchmark. However, SimLingo relies on access to “action dreaming” data. This consists of safe and unsafe trajectories collected in CARLA with access to privileged information, exposing the policy to counterfactual scenarios beyond expert demonstrations. SteerVLA’s auto-labeling pipeline does not require access to privileged simulation information, and can be easily transferred to real-world data. We find that we can achieve improved long-tail performance with the labels generated by our auto-labeling pipeline in combination with a hierarchical policy structure. We also do not rely on additional data to improve reasoning and steerability, but achieve this through our hierarchical architecture and detailed meta-action labels to steer the low-level policy, especially in scenarios where reasoning about other agents is required, such as the “blocked intersection” and “construction zone” scenarios shown in Fig. 4 and Fig. 5 . Data labeling for Autonomous Driving. Several works have aimed to label driving data with language to improve interpretability and reasoning capabilities in driving models. Some use a purely manual labeling process ( 50 ; 11 ; 29 ; 49 ) , which introduces high overhead, but can result in more natural and realistic labels. Other works use a mixture of VLM-generated labels and human verification to create a more scalable labeling pipeline ( 40 ; 23 ; 17 ) . SteerVLA is trained on data generated from our fully automatic labeling pipeline and does not require human supervision. Many of these works focus on visual question-answering tasks ( 40 ; 23 ) or explanations of driving behavior ( 50 ; 29 ) , aiming to improve scene understanding and interpretability but with limited focus on precisely prescribing the actions the vehicle should take. SteerVLA’s auto-labeling pipeline aims to extract information from driving data and organize it into comprehensive reasoning traces and detailed meta-action labels. These labels go beyond the typical commands used in prior work, such as “Accelerate” or “Turn left”. Instead, we augment these commands with the manner in which these behaviors should be executed, for example, “Accelerate cautiously with a slight left adjustment to avoid the construction site”. To further enhance the VLM’s spatial reasoning during labeling, we overlay mid-level representations (bounding boxes or trajectory projections) onto the driving images. By training the high-level policy to generate these information-rich meta-actions and the low-level policy to follow them, we can effectively ground the high-level reasoning into low-level control. 3 Preliminaries Figure 2 : Framework overview . Our model reasons over the driving context to produce reasoning traces and fine-grained meta-actions that guide driving control. A VLM generates structured semantic guidance, which steers a VLA policy in predicting future waypoints. To supervise reasoning and meta-action generation, we introduce an automatic data labeling pipeline that derives fine-grained language supervision from driving trajectories, improving alignment between language and control. Problem statement. We formulate autonomous driving as a sequential decision-making problem. We assume access to a standard navigation system that provides high-level routing commands to guide the vehicle toward a destination. At each timestep t , the agent receives an observation o_{t}=\{I_{t},q_{t-k+1:t}\} and a routing command \ell_{t} , where I_{t} is the current front-view camera image, q_{t-k+1:t} denotes the recent history of ego vehicle states (e.g., past speeds and headings over the last k steps), and \ell_{t} is provided by a navigation system (e.g., turn-by-turn guidance such as “turn left in 50 m”). The objective is to predict a chunk of future actions A_{t}=[a_{t},a_{t+1},\ldots,a_{t+H-1}] that specify low-level control signals (e.g., future waypoints) over a horizon H ( 57 ) . A driving policy is therefore a conditional distribution \pi(A_{t}\mid o_{t},\ell_{t}) that maps the current observation and routing command to a distribution over action chunks. Training proceeds by maximizing the likelihood of expert demonstrations: \max_{\theta}\ \mathbb{E}_{(A_{t},o_{t},\ell_{t})\sim D}\,[\log\pi_{\theta}(A_{t}\mid o_{t},\ell_{t})] , where D is a dataset of expert driving trajectories paired with synchronized routing commands. Driving vision-language-action models. Recent driving VLA models ( 22 ; 59 ) learn a direct mapping from routing commands \ell_{t} and visual observations I_{t} to driving control actions A_{t} , typically by fine-tuning a pretrained vision–language model on driving data. Some approaches, such as SimLingo ( 37 ) , generate a language-based description of the intended driving behavior (i.e., a meta-action) as a chain-of-thought step before producing driving actions. Our low-level policy in CARLA builds upon SimLingo ( 37 ) , which uses InternVL2-1B ( 10 ) as the pretrained VLM backbone and represents actions as future waypoints. Waypoint prediction is performed by lightweight MLP heads on top of the VLM outputs and is trained using a Smooth L1 loss. 4 SteerVLA In this section, we introduce SteerVLA, a framework that leverages VLM semantic reasoning to steer a VLA policy toward grounded and context-aware driving control. An overview of SteerVLA is shown in Fig. 2 . 4.1 Steering VLA Control using VLM Reasoning We focus on the challenge of long-tail driving scenarios, where rare and unanticipated events require strong generalization and common-sense reasoning from the policy. VLAs are a strong backbone for driving because they combine semantic grounding from vision–language pretraining with domain-specific adaptation obtained via imitation learning on driving data. Building on this capability, we leverage the reasoning and semantic inference abilities of VLMs and ground these inferences in driving control through fine-grained meta-actions that steer a VLA policy. Concretely, a high-level policy first reasons about the driving scene, historical vehicle states, and routing command to produce a meta-action m_{t} , accompanied by a short reasoning trace c_{t} that reasons over driving scenes and helps the policy generate more appropriate meta-actions. Formally, given observation o_{t}=\{I_{t},q_{t-k+1:t}\} with image I_{t} , historical ego vehicle states q_{t-k+1:t} , and routing command \ell_{t} , the high-level policy outputs (c_{t},m_{t})\sim\pi_{hl}(c_{t},m_{t}\mid o_{t},\ell_{t}) . The low-level VLA then predicts future waypoints A_{t}=[a_{t},a_{t+1},\ldots,a_{t+H-1}] conditioned on both the observation and the meta-action, i.e., A_{t}\sim\pi_{ll}(A_{t}\mid o_{t},m_{t}) . This design improves generalization by offloading high-level reasoning to the high-level policy, while allowing the VLA to specialize in fast and accurate waypoint prediction conditioned on the high-level’s instructions. High-level policy. We finetune the high-level policy \pi_{hl}(c_{t},m_{t}|o_{t},\ell_{t}) with a pre-trained VLM as the base model. Our dataset generation pipeline, which we introduce in Section 4.2 , provides supervision for the high-level policy outputs. The VLM’s strong semantic priors allow the high-level policy to reason about the vehicle’s surroundings and encode rich contextual information into its predicted meta-actions, enabling more context-aware action predictions. We structure the query to the VLM as a visual question-answering problem by providing the current visual observation I_{t} , a six-second-long history of ego states q_{t-k+1:t} (speed and heading) sampled at 0.5 Hz where k=3 , and a routing command \ell_{t} . We train the model via a next-token prediction objective to generate a chain-of-thought reasoning trace c_{t} describing the positions and movement of critical agents in the scene, followed by an appropriate meta-action m_{t} . Low-level VLA policy. Instead of using a general routing command as language input, our low-level policy \pi_{ll}(A_{t}|o_{t},m_{t}) is steered by fine-grained meta-actions m_{t} generated by the high-level policy \pi_{hl}(c_{t},m_{t}|o_{t},\ell_{t}) (see Section 4.2 for details on generating meta-action labels). We do not provide the reasoning trace as input to the low-level policy. The high-level policy already uses the reasoning trace as chain-of-thought to synthesize a well-grounded meta-action that distills the contextual and semantic information required for control. Conditioning the low-level policy only on the meta-action allows it to focus on precise execution and instruction following rather than semantic reasoning. 4.2 Generating language labels for SteerVLA To address the scarcity of fine-grained, control-grounded language supervision in driving datasets, we develop a fully autonomous labeling pipeline. We perform a two-stage query to Gemini 2.5 Flash-Lite ( 43 ) that first identifies the baseline action taken by the vehicle (e.g., changing lanes, continuing straight), then uses trajectory information to enrich the language labels with fine-grained behavioral details explicitly tied to driving control. First, in our baseline categorization query, we provide Gemini 2.5 Flash-Lite with a grounded representation of the vehicle’s action using a projection of the vehicle’s future trajectory over a front-facing camera view. We then perform a refinement step by providing the VLM with the vehicle’s speed and course over time to produce a nuanced description of the vehicle’s action. For example, we transform the original label “the car is continuing straight” into the more fine-grained “the car normally accelerates, then maintains speed while subtly drifting right”. This refinement step is crucial for passing as much information as possible to the low-level policy and can be applied to any existing language-labeled driving dataset, allowing us to augment these data with additional information that can improve steerability and performance of the driving policy. To improve the reasoning capabilities of the high-level policy, we additionally generate reasoning trace labels for each trajectory in the training data. These traces describe the scene and characterize the motion of other agents, serving as chain-of-thought supervision that guides the prediction of meta-actions. More details on the prompts used for our auto-labeling pipeline are provided in Section B.1 of the appendix. 4.3 Implementation Details For closed-loop evaluation in the CARLA simulator, we follow the recipe from SimLingo ( 37 ) to train the driving policy, building both the high-level and low-level policies upon InternVL2-1B ( 10 ) as the pretrained VLM. Future actions are represented by two types of waypoints: (1) time-based waypoints sampled at 4 Hz, which determine target speed and temporal motion, and (2) geometry-based waypoints sampled at 1 m intervals, which describe the planned path and guide steering. These waypoints are converted into driving controls via PID controllers. We apply the refinement and reasoning trace generation processes described in Section 4.2 to the SimLingo dataset. To construct reasoning trace labels for the high-level policy, we leverage available 3D bounding box annotations to identify agents within the ego vehicle’s field of view. We then transform relevant attributes, such as velocity and position relative to the ego vehicle, into structured natural language descriptions (e.g., “Red car, in one lane to the left, traveling same direction, at 6.1 m/s.”). We synchronize the high- and low-level policies by querying them at the same frequency. At each step, the low-level policy waits for the high-level to generate a meta-action, which it then uses as language input to produce driving actions. Specifically, we query the framework at 20 Hz in the CARLA simulator. 5 Experimental Results Figure 3 : Closed-loop evaluation of SteerVLA on Bench2Drive. On Bench2Drive, we report overall performance and per-ability scores for SteerVLA across five advanced urban driving skills. SteerVLA significantly outperforms prior approaches, benefiting from improved reasoning and instruction-following capabilities. Figure 4 : Closed-loop evaluation of SteerVLA on Bench2Drive-LongTail. We compare SteerVLA with the state-of-the-art method SimLingo on Bench2Drive-LongTail. SteerVLA exhibits larger performance gains in long-tail scenarios, likely because these cases require more complex reasoning and more precise control. Our experiments are designed to address the following key research questions: RQ1: How does SteerVLA perform in simulated closed-loop driving under diverse traffic conditions, particularly in long-tail scenarios? RQ2: How effectively does SteerVLA reason about complex scenes and follow driving instructions (i.e., meta-actions)? RQ3: How effective is each component in SteerVLA? RQ4: Can SteerVLA generalize to real-world driving data? 5.1 Experimental Setup The majority of our experiments use the CARLA simulator to perform large-scale closed-loop evaluation of SteerVLA under diverse driving conditions. We evaluate SteerVLA on the Bench2Drive ( 25 ) benchmark, which contains 220 driving scenarios in 12 towns, including adverse weather and lighting conditions, such as fog, nighttime driving, and various long-tail driving scenarios, such as construction sites, traffic accidents ahead, and jaywalking pedestrians. We also construct a benchmark Bench2Drive-LongTail consisting of a long-tail scenario subset from Bench2Drive, described in Section 5.2 with additional detail in Section C.2 , to further study the performance of SteerVLA on long-tail scenarios. Training dataset. We use the driving trajectories and meta-action labels from the SimLingo ( 37 ) dataset, and further apply our data augmentation pipeline to generate reasoning traces and refine the meta-actions into more fine-grained descriptions. We train both the high- and low-level policies on this dataset. Policy deployment. We run SteerVLA at 20 Hz in the CARLA simulator. On a single NVIDIA L40 GPU, SteerVLA incurs an inference latency of 2.51 s. As our closed-loop evaluation is conducted entirely in simulation, we do not optimize inference efficiency in this work. For real-world deployment, we plan to reduce inference latency using standard acceleration techniques, such as KV caching, in future work. Evaluation metrics. We evaluate closed-loop performance using the driving score, following the official CARLA metric. Driving score jointly measures task completion and safety by combining route completion with penalties for traffic infractions. Specifically, for each route, the route completion percentage is multiplied by penalties corresponding to the severity of infractions incurred. This metric captures both driving progress and robustness to safety violations. Baselines. We evaluate several recent vision-language-action (VLA) baselines. Full descriptions of these methods are provided in Section C.1 . We compare SteerVLA to SimLingo ( 37 ) , the current top method on the CARLA 2.0 Leaderboard trained with counterfactual data to improve language following, DriveMoE ( 52 ) , a mixture-of-experts method, an alternative to the hierarchical structure we introduce, ORION ( 15 ) , an end-to-end method that focuses on long-term history aggregation and improves driving reasoning with question-answering as a co-training task rather than keeping a distinct high-level policy to maintain reasoning capabilities, and AutoVLA ( 61 ) , which uses a pretrained VLM with a physical action codebook. These methods present alternative methods to retain reasoning capabilities or improve reasoning for driving tasks from a pretrained VLM. Method Architecture Language Labels Driving Score \uparrow SimLingo VLA Meta-actions 85.94 SteerVLA VLM-VLA Meta-actions 88.81 SteerVLA VLM-VLA Refined meta-actions + Reasoning traces 90.71 Table 1: Ablation study of SteerVLA components on Bench2Drive. Our results show that SteerVLA benefits substantially from grounded semantic reasoning and fine-grained meta-actions produced by a high-level policy that effectively steers low-level control. This is enabled by our data labeling pipeline, which aligns fine-grained meta-action and reasoning supervision with low-level control signals extracted from driving trajectories. (a) Comparison of SteerVLA with off-the-shelf VLM in reasoning capability. We compare the reasoning capability in SteerVLA high-level policy with Gemini 2.5 Flash Lite (prompting details in Section B.4 ). When faced with a long-tail scenario, the off-the-shelf VLM (left) can roughly reason about the state of the scene but struggles to reason about the immediate actions the ego-vehicle should take under current conditions. Conversely, SteerVLA (right) produces both descriptive reasoning and can aptly generate a meta-action to navigate around the construction site. (b) Comparison of SteerVLA with SimLingo in reasoning capability. The detailed reasoning traces and meta-actions used to train SteerVLA enable flexible environmental inference and timely decision-making when a nearby vehicle does not decelerate to give way during a lane change, whereas SimLingo fails to generate timely meta-actions and collides with another vehicle. (c) Comparison of SteerVLA with SimLingo in instruction following capability. After turning right into a new lane where the vehicle ahead blocks the path, both methods output the meta-action decelerate to follow the blue vehicle. SteerVLA decelerates immediately, while SimLingo’s delayed deceleration leads to a collision. Green points show predicted future waypoints sampled at 4 Hz. Figure 5 : Long-tail scenario case study. We analyze how SteerVLA reasons and acts in long-tail driving scenarios. In (a) , we compare SteerVLA with an off-the-shelf VLM, showing that SteerVLA produces both descriptive reasoning and actionable meta-actions. In (b) , we compare SteerVLA with SimLingo in a lane-change interaction where an adjacent vehicle does not give way, highlighting SteerVLA’s ability to make timely high-level decisions. In (c) , we evaluate instruction-following behavior when the lane ahead is blocked, where SteerVLA executes the deceleration meta-action immediately while SimLingo exhibits delayed control. 5.2 Evaluating SteerVLA on Driving Performance Towards answering Q1 , we evaluate SteerVLA closed-loop on the Bench2Drive benchmark. We additionally extract a subset of routes from Bench2Drive to observe the long-tail reasoning capabilities of SteerVLA and its ability to act appropriately in these scenarios. Results are shown in Fig. 3 and in Fig. 4 with a detailed table of results in Section C.3 . We also include discussion on failure cases in Section C.4 . Driving performance on Bench2Drive. Fig. 3 demonstrates that SteerVLA has strong performance on the Bench2Drive benchmark, achieving a better driving score than the next best baseline, SimLingo, by 4.77, and outperforms it in all ability categories, except merging. We observe that SteerVLA tends to outperform SimLingo in highly dynamic scenarios (i.e., merging into the oncoming lane around a construction zone, or navigating an accident on the road). SteerVLA’s reasoning trace structure guides the policy to make conjectures about the movement intent of the other agents within the scene, enabling SteerVLA to prepare and preemptively react to adversarial behavior. This is also made apparent through the case study presented in Fig. 5 . Driving performance in long-tail scenarios. On Bench2Drive-LongTail, SteerVLA shows clear advantages over the strongest baseline, SimLingo. Fig. 4 demonstrates that SteerVLA, on average, outperforms SimLingo by 8.04 in driving score, with especially large deltas in the construction zone, traffic accident, and blocked intersection scenarios. Long-tail scenarios demand rich semantic reasoning and precise control. In these scenes, inferences about out-of-distribution objects and the behavior of other agents must be carefully interpreted to navigate the changes in traffic flow from typical scenarios. SteerVLA is able to address this by explicitly reasoning about the states of other agents and traffic signs, resulting in detailed meta-actions that can guide VLA to cautiously but decisively act when it is safe to do so. 5.3 Case Study in Long-Tail Scenarios Towards answering Q2 , we perform case studies of SteerVLA on challenging long-tail scenarios to observe its reasoning and instruction following capabilities, shown in Fig. 5 . We first compare SteerVLA’s high-level policy reasoning to Gemini 2.5 Flash Lite, the same VLM we use in our autolabeling pipeline. In Fig. 5(a) , when faced with a construction site blocking a lane, while the off-the-shelf VLM can roughly reason about the scene state, it struggles to reason about the immediate actions the ego-vehicle should take. In contrast, SteerVLA produces both descriptive reasoning and aptly generates a meta-action to navigate around the construction site. Comparing to SimLingo in Fig. 5(b) , we observe that the detailed reasoning traces and meta-actions used to train SteerVLA enable more dynamic, information-rich reasoning and timely decision-making. When a nearby vehicle does not decelerate during a lane change, SteerVLA generates appropriate meta-actions by reasoning about other agents and traffic conditions, while SimLingo fails to react in time and collides. Furthermore, in Fig. 5(c) , when both methods correctly output a deceleration meta-action to follow a blocking vehicle, SteerVLA executes immediately while SimLingo’s delayed response leads to a collision. Overall, these case studies demonstrate that the reasoning traces and meta-actions of SteerVLA enable state-of-the-art long-tail scenario performance through superior environmental reasoning and precise action timing. 5.4 Ablations of SteerVLA Towards answering Q3 , we study the effectiveness of two key design choices in SteerVLA on Bench2Drive: the VLM-VLA style architecture and the reasoning components (i.e. fine-grained meta-actions and reasoning traces). We evaluate the effectiveness of a hierarchical architecture by comparing SteerVLA with SimLingo under the same meta-action supervision from the SimLingo dataset. We assess the impact of fine-grained meta-actions and reasoning traces by comparing SteerVLA trained with refined language labels generated by our data labeling pipeline against training with the original meta-action labels from SimLingo. The results in Table 1 show that SteerVLA benefits substantially from grounded semantic reasoning and fine-grained meta-actions produced by the high-level policy that effectively steer low-level control. This is enabled by our data labeling pipeline, which aligns fine-grained meta-action and reasoning supervision with low-level control signals extracted from driving trajectories. 5.5 Open-Loop Evaluation on Real-World Data Towards answering Q4, we additionally evaluate SteerVLA in an open-loop setting on the NuScenes benchmark ( 7 ) , using stronger VLM backbones—Gemma3-4B ( 44 ) as the high-level policy and PaliGemma-3B ( 3 ) as the low-level policy. We adopt more powerful pretrained VLMs for this evaluation, as real-world driving scenarios require stronger visual semantic generalization and benefit from richer prior knowledge learned from large-scale real-image data. Besides, since NuScenes does not provide language annotations, we apply our auto-labeling pipeline to generate grounded meta-actions and reasoning traces directly from raw driving data. We emphasize that our primary evaluation focuses on closed-loop results on Bench2Drive, which provide a more meaningful assessment by capturing control dynamics, recovery behavior, and long-horizon interactions with the environment, whereas open-loop evaluation on NuScenes measures trajectory imitation on fixed data. SteerVLA achieves comparable performance to existing methods on the NuScenes benchmark. Detailed experimental settings and results on NuScenes are provided in Section C.5 . 6 Discussion We presented SteerVLA, a hierarchical VLA model for autonomous driving. Our approach decomposes driving into a high-level, language-based reasoning step and a low-level action generation step, with detailed meta-actions serving as the interface between the two. These meta-actions are generated from driving data through a fully automatic labeling pipeline. As a result, SteerVLA can reason over complex driving scenarios and produce precise control outputs, achieving state-of-the-art performance in both general and long-tail driving settings. While SteerVLA demonstrates improved reasoning and steerability, it has several limitations that point to important future directions. The quality of our auto-labeling pipeline is constrained by the capabilities of the underlying VLM, particularly for temporally grounded understanding in video contexts. Our current system also uses only a single camera view, limiting scene coverage; extending to multi-view camera inputs would enhance spatial awareness and better match real-world autonomous vehicle sensor configurations. Overall, we hope that our work represents a step toward real-world systems that can use common sense to deal with complex and unfamiliar situations. We expect that the capabilities of VLMs and other foundation models will continue to improve, providing better multi-modal reasoning in diverse scenarios, and grounding these capabilities in real-world actions would allow for increasingly robust autonomous systems. SteerVLA represents a step toward this future. Appendix A Training Details A.1 Model Architecture While our method is applicable to any VLM backbone, we use InternVL2-1B ( 10 ) for both the high-level and low-level policies in our closed-loop experiments. InternVL2-1B is based on Qwen2.5-0.5B-Instruct ( 36 ) and uses InternViT-300M-448px as its vision encoder. Following the design in ( 37 ) , our low-level policy employs two additional MLP heads for future waypoint prediction. We fine-tune the language model with LoRA and apply full fine-tuning to all remaining parameters. A.2 Training Hyperparameters Hyperparameters are shown in Table 2 . High-Level Policy Hyperparameter Value Batch Size 96 Gradient Accumulation Steps 2 Epochs 20 Learning Rate 3\times 10^{-5} Learning Rate Scheduler Cosine Decay Betas (0.9,0.999) Optimizer AdamW Warmup steps 5% of total steps LoRA alpha 64 LoRA r 32 LoRA dropout 0.1 Low-Level Policy Hyperparameter Value Batch Size 120 Gradient Accumulation Steps 1 Epochs 30 Learning Rate 3\times 10^{-5} Learning Rate Scheduler Cosine Decay Betas (0.9,0.999) Optimizer AdamW Warmup steps 5% of total steps LoRA alpha 64 LoRA r 32 LoRA dropout 0.1 Table 2 : Training Hyperparameters. A.3 Training and Inference Hardware We trained our high-level policy on 8 NVIDIA H100 GPUs for 15 hours and our low-level policy on 4 NVIDIA H200 GPUs for 20 hours. Inference was performed on a single NVIDIA L40 GPU. Appendix B Auto-Labeling Pipeline Details B.1 Label Refinement on SimLingo dataset In order to imbue language labels from language-annotated datasets, such as the SimLingo dataset, with detailed movement information as described in Section 4.2 , we provide the vehicle’s ego states over a period of three seconds in addition to the original language label in a prompt (see Listing for the full prompt) to Gemini 2.5 Flash-Lite. SimLingo Refinement Prompt. ⬇ You are an expert in vehicle dynamics and driving behavior analysis . Your task is to interpret and refine natural language descriptions of driving behavior by analyzing vehicle ego state data ( speed and course over time ) to produce a ** precise and nuanced behavior summary **. Your output should describe : 1. ** Ego State Analysis ** - a brief explanation of observed speed and course trends over time . 2. ** Refined Driving Behavior Description ** - a more specific version of the original description , enhanced with a meaningful modifier _ ( e . g ., ** smooth turning **, ** wide turn **, ** abrupt stop **, ** steady lane keeping **) _ and a ** driving style **, reflecting the driver ’ s attitude or intent _ ( e . g ., ** cautiously **, ** normally **, ** aggressively **) _ --- ## Input Format ** Driving Description :** { commentary } ** Ego Vehicle State Sequence ** ( next 3 seconds from frame { frame_number }): { ego_states_sequence } These ego states reflect how the vehicle moved during the described behavior . > ** Note :** > - ** Course increasing ** -> vehicle is adjusting ** right ** > - ** Course decreasing ** -> vehicle is adjusting ** left ** --- ## Output Guidelines Your response should contain two sections : ### 1. Ego State Analysis Analyze the speed and course sequence : - Describe speed patterns : Is the vehicle accelerating , decelerating , or maintaining speed ? - Describe course patterns : Is the vehicle turning sharply , smoothly , or going straight ? - Mention time duration and total changes in course or speed . ### 2. Refined Driving Behavior Description Produce a single , natural - language sentence that : - Refines the driving description with motion extent ( e . g ., * smooth *, * sharp *, * wide *, * slight *) - Adds driving style ( e . g ., * cautiously *, * normally *, * aggressively *) - Grounds the refinement in the observable patterns of the ego vehicle states - Do not change the semantic meaning of the original description . Only use the ego states to refine the description . --- ## Notes - The refined description must not exceed **20 words **. - Use ** speed trends ** to judge acceleration or deceleration patterns . - Use ** course change patterns ** to assess turning sharpness or trajectory smoothness . - If the style cannot be confidently inferred , default to **" normally "**. - Use ** natural , human - readable language **- avoid unnecessary technical jargon . - The refined description must be a single sentence in present tense and third person ( i . e . " The vehicle turns ..." or " The car accelerates ...") - If the driving description includes any references to external vehicles , pedestrians or traffic constructs , maintain this information in the final refined description , as well as their distances from the ego vehicle and any descriptiors ( i . e . color ) if available . - Do not change the semantic meaning of the original description . Only use the ego states to refine the description . - If the original description mentions specific maneuvers , i . e . lane changes , retain this information . - Unless a turn is explicitly mentioned in the original description , heading changes of 30 degrees or below should be described as ** adjustments ** to the left or right , and not turns . B.2 NuScenes Meta-Action Labeling Our auto-labeling pipeline is also applicable to real-world datasets without prior language labels. To apply our labeling pipeline to the NuScenes dataset, we begin by splitting trajectories into 2-5 second chunks based on a set of heuristics that define the boundaries of where a specific category of action (e.g., accelerating, changing lanes, or turning) is likely to have occurred. Specifically, we apply a 1D Gaussian blur to the vehicle’s speed and course changes over time, and apply splits where the vehicle is stopped, or the vehicle’s acceleration or angular velocity are above certain thresholds for an extended period of time. We then utilize the vehicle’s camera extrinsic and intrinsic matrices to produce a projection of the vehicle’s future trajectory over front camera views from the first and middle frames of the trajectory. These images, the vehicle’s ego states and lane IDs over time, and the prompt in Listing are provided to Gemini 2.5 Flash-Lite for a baseline categorization stage. We show two examples in Fig. 6 . Example Meta Action Labeling Prompt. ⬇ You are an expert in vehicle dynamics and driving behavior analysis . You have been provided two frames from a dashcam video from a vehicle , with a projected green , yellow , and red trajectory overlaid on the first and middle frames of the video of the trajectory that the vehicle is in the process of taking . The images are labelled " First Frame " and " Middle Frame " at the tops of the images . Describe : 1. Ego State Analysis : Analyze the speed and course sequence : - Describe speed patterns : Is the vehicle accelerating , decelerating , or maintaining speed ? - Describe course patterns : Is the vehicle turning sharply , smoothly , or going straight ? - Mention time duration and total changes in course or speed . These ego states reflect how the vehicle moved during the described behavior . > ** Note :** > - ** Course increasing ** -> vehicle is moving ** right ** > - ** Course decreasing ** -> vehicle is moving ** left ** { ego_states_text } 2. First frame description : - Describe the lane markings in the first frame image , and the projected trajectory ’ s position relative to them at the beginning of the trajectory and at the end . Identify any areas on the road with solid white or yellow lines . - Are there road markings , signs , or other structures that indicate that the vehicle is at an intersection ? - Which lane does the trajectory begin in , and which lane does the trajectory end in ? - Is the red , yellow , and / or green trajectory to the right or left of the lane markings ? - Is the cyan circle to the right or left of the lane markings ? - Is the trajectory curving ? If so , which way is the trajectory curving ? 3. Middle frame description : - Describe the lane markings in the middle frame image , and the projected trajectory ’ s position relative to them at the beginning of the trajectory and at the end . Identify any areas on the road with solid white or yellow lines . - Are there road markings , signs , or other structures that indicate that the vehicle is at an intersection ? - Which lane does the trajectory begin in , and which lane does the trajectory end in ? - Is the red , yellow , and / or green trajectory to the right or left of the lane markings ? - Is the cyan circle to the right or left of the lane markings ? - Is the trajectory curving ? If so , which way is the trajectory curving ? 4. Consolidated Analysis : - Based on your analysis of the first frame image and the middle frame image , which lane does the vehicle begin in , and which lane does it end in ? - Does this signify a lane change ? If so , is the vehicle making a lane change to the left , or a lane change to the right ? - Alternatively , is the vehicle at an intersection in either frame ? Does this signify a turn ? Even if the trajectory is curving , consider whether the course change is large enough to be a turn , and whether the vehicle is simply continuing forward to a parallel road . - If so , is the vehicle turning to the left , or to the right ? 5. Vehicle Action : The action that the vehicle is taking . Is the vehicle ** turning **, ** changing lanes **, or ** continuing straight **? If the vehicle is turning or changing lanes , is it doing so to the ** left ** or to the ** right **? Choose from one of the following discrete actions : - turning left - turning right - changing lanes left - changing lanes right - continuing straight - completely stationary - making a U - Turn Notes : - The cyan circle denotes the ** end ** of the trajectory . - The trajectory begins at the ** bottom ** of the image . - A turn is defined as a full turn at an intersection . - Otherwise , if the trajectory is simply following a curve in the road , describe this as ** continuing straight ** - If the trajectory is ** continuing straight ** through an intersection , describe this as ** continuing straight ** - If the vehicle has crossed a lane marking , it is most likely making a lane change . - There may be no visible trajectory projected , in which case the vehicle is most likely moving very slowly or stationary . - Identify only the lane markings that are clearly discernible . - Small course changes of fewer than 4 degrees most likely indicate that the vehicle is ** continuing forward **. - Large course changes over 50 degrees likely indicate that the vehicle is ** turning **. - Small velocities below 1.0 meters per second likely indicate that the vehicle is stationary . Lane information : { lane_information } We then provide the output of Listing , the vehicle’s ego states, and the prompt in Listing to Gemini 2.5 Flash-Lite in a refinement step that imbues the resulting meta-action with more detailed movement information. Example Meta Action Labeling Prompt. ⬇ # Driving Behavior Refinement Prompt You are an expert in vehicle dynamics and driving behavior analysis . Your task is to interpret and refine natural language descriptions of driving behavior by analyzing vehicle ego state data ( speed and course over time ) to produce a ** precise and nuanced behavior summary **. Your output should describe : 1. ** Ego State Analysis ** -> a brief explanation of observed speed and course trends over time . 2. ** Refined Driving Behavior Description ** - a more specific version of the original description , enhanced with a meaningful modifier _ ( e . g ., ** smooth turning **, ** wide turn **, ** abrupt stop **, ** steady lane keeping **) _ and a ** driving style **, reflecting the driver ’ s attitude or intent _ ( e . g ., ** cautiously **, ** normally **, ** aggressively **) _ --- ## Input Format ** Driving Description :** { driving_description } ** Ego Vehicle States :** { ego_state_sequence } These ego states reflect how the vehicle moved during the described behavior . > ** Note :** > - ** Course increasing ** -> vehicle is moving ** right ** > - ** Course decreasing ** -> vehicle is moving ** left ** --- ## Output Guidelines Your response should contain two sections : ### 1. Ego State Analysis Analyze the speed and course sequence : - Describe speed patterns : Is the vehicle accelerating , decelerating , or maintaining speed ? - Describe course patterns : Is the vehicle turning sharply , smoothly , or going straight ? - Mention time duration and total changes in course or speed . ### 2. Refined Driving Behavior Description Produce a single , natural - language sentence that : - Refines the driving description with motion extent ( e . g ., * smooth *, * sharp *, * wide *, * slight *) - Adds driving style ( e . g ., * cautiously *, * normally *, * aggressively *) - Grounding the refinement in the observable patterns of the ego vehicle states --- ## Notes - The refined description must not exceed **20 words **. - Use ** speed trends ** to judge acceleration or deceleration patterns . - Use ** course change patterns ** to assess turning sharpness or trajectory smoothness . - If the style cannot be confidently inferred , default to **" normally "**. - Use ** natural , human - readable language **- avoid unnecessary technical jargon . - If the driving description is " The vehicle is continuing straight ", describe any left or right movements as " adjusting left " or " adjusting right " respectively . Do not describe this as turning . (a) Lane change left example (b) Continue straight example Figure 6 : Input images for meta-action labeling. The first-round prompt gives a baseline action and the second-round prompt produces a refined meta-action. (a) Example where the baseline action is “changing lanes left,” refined to “The vehicle is smoothly changing lanes left normally.” (b) Example where the baseline action is “continuing straight,” refined to “The car normally accelerates, then maintains speed while subtly drifting right.” B.3 NuScenes Reasoning Labeling To produce reasoning traces for the NuScenes dataset, we provide a front camera view, as well as the prompt in Listing to Gemini 2.5 Flash-Lite. Example Reasoning Labeling Prompt. ⬇ You are an expert in autonomous driving planning . Given a first person dashcam view from a car and the car ’ s future action , describe the following : Future action : { meta_action } 1. Provide a sentence of justification for the car ’ s future action . - A concrete example is as follows : There is a car in the oncoming lane and an accident ahead of me , so I should wait within my lane until the oncoming car is clear . 2. Behavior description of critical objects : describe the current status and intent for the 2-3 most important critical objects in the image ( e . g . pedestrians , vehicles , cyclists , stop signs , traffic lights , construction cones , etc .) in 3 sentences or fewer . - A concrete example is as follows : The pedestrian is currently standing on the sidewalk , looking toward the road , and maybe preparing to cross the street . The vehicle is currently ahead of me , moving in the same direction , and its future trajectory suggests it will continue straight . B.4 VLM Zero-shot Prompt To compare the zero-shot capabilities of an off-the-shelf VLM on our meta-action prediction task, we provide the prompt in Listing to Gemini 2.5 Flash-Lite (the same model used for labeling), together with the vehicle’s visual observation. Zero-shot Prompt Provided to Gemini 2.5 Flash-Lite. ⬇ You are an autonomous driving assistant . Your task is to generate a driving behavior plan based on : A front - view camera image A sequence of historical ego states taken at 0.5 Hz over the past 6 seconds The current speed of the vehicle A routing command . Inputs : Image : < first person image from a dashcam view > Speed history : 0.0 m / s 0.0 m / s 0.0 m / s Heading history : 165.8 degrees 165.8 degrees 165.8 degrees Current speed : 0.0 m / s Routing command : Follow the road . Output : 1. Behavior description of critical objects : describe the current movement and appearance of all external agents in the scene , as well as their positioning relative to the ego vehicle . Example Output : " Red car , in one lane to the left , traveling same direction , at 6.1 m / s . Female pedestrian , in crosswalk , travelling opposite direction , at 2.1 m / s ." 2. Driving Behavior Plan : Produce a driving behavior plan ( no more than 20 words ) that includes : Speed behavior - Will the vehicle accelerate , maintain speed , or decelerate ? Heading behavior - Describe the expected heading change ( e . g ., continue straight , turn slightly right , make a sharp left ). Driving style - Reflect the style ( e . g ., cautiously , smoothly , assertively ). Respond with a single natural language sentence summarizing the driving behavior . Example Output : " The car decelerates smoothly and makes a slight right turn , driving normally to follow the blue SUV ." Notes : - The driving behavior plan must be in present tense and third person ( i . e . " The vehicle ...") Appendix C Experiment Details C.1 Full description of baselines SimLingo ( 37 ) . A vision-only VLM framework that addresses closed-loop driving, vision-language understanding, and language-action alignment, relying solely on cameras and avoiding costly sensors such as LiDAR. SimLingo additionally leverages “action-dreaming” data, which is counterfactual data used to improve its language following capabilities. SimLingo is currently the top method on the CARLA 2.0 leaderboard. DriveMoE ( 52 ) . Built upon the \pi_{0} foundation model ( 4 ) , DriveMoE employs a mixture-of-experts architecture with a scene-specialized vision MoE and a skill-specialized action MoE to achieve adaptive decision making for autonomous driving. ORION ( 15 ) . A holistic E2E framework that integrates a QT-Former for long-term history aggregation, an LLM for driving scenario reasoning, and a generative planner for precise trajectory prediction. ORION further aligns reasoning and action spaces, enabling unified optimization across both planning and visual question answering, though at the cost of greater complexity and computational demand. AutoVLA ( 61 ) . A method that enhances a pretrained VLM with a physical action codebook for vehicle motion, effectively bridging semantic reasoning and low-level control. PARA-Drive ( 48 ) . A modular end-to-end autonomous driving model that uses bird’s-eye-view features and is parallelized to improve runtime efficiency, which offers a comparison alternative modular architecture to our hierarchical structure for comparison. TOKEN ( 45 ) . A method that tokenizes sensory inputs into object-centric tokens using an end-to-end driving model, PARA-Drive, trained with various driving tasks to enforce good representations. This method leverages explicit structure inspired by traditional driving stacks rather than leveraging VLM priors to make good driving decisions. DiMA-VAD ( 18 ) . DiMA-VAD distills knowledge from a VLM into a driving model through jointly training the VLM and a vision-based planner on a set of surrogate driving understanding and prediction tasks rather than directly using the VLM as a base model. Agent-Driver ( 31 ) . This method decomposes the LLM’s tasks into using a tool library to process sensory inputs, structuring information in a memory module, and performing motion planning with a reasoning engine. This method explicitly performs many of the reasoning steps that are implicitly included in our auto-labeling pipeline. Method Sensors DS \uparrow SR(%) \uparrow Ability \uparrow Merging Over- taking Emergency Brake Give Way Traffic Sign Mean DriveMoE M 74.22 48.64 34.67 40.00 65.45 40.00 59.44 47.91 ORION M 77.74 54.62 25.00 71.11 78.33 30.00 69.15 54.72 AutoVLA M 78.84 57.73 - - - - - - SimLingo S 85.94 66.82 57.50 60.00 76.67 50.00 73.16 63.46 SteerVLA (Ours) S 90.71 73.64 56.25 84.44 81.67 60.00 81.05 72.68 Table 3 : Evaluation of SteerVLA on Bench2Drive. Metrics include Driving Score (DS), Success Rate (SR%), and specialized abilities (Merging, Overtaking, Emergency Brake, Give Way, Traffic Sign) with overall Mean performance. Compared to the state-of-the-art, SteerVLA outperforms the best performing baseline (SimLingo). M/S refers to Multi-camera/Single camera. Driving score \uparrow Long-tail Scenario # Routes Simlingo SteerVLA Illegally Parked Vehicle 10 96.00 100.00 Adjacent Door Opening 5 92.00 92.00 Roadside Cyclist 10 88.00 88.00 Construction Zone 10 62.66 96.50 Traffic Accident 10 68.91 86.33 Jaywalking Pedestrian 10 100.00 95.00 Crossing Vehicle Runs Red 5 71.94 72.06 Control Loss 5 100.00 100.00 Hard Brake 5 100.00 98.86 Blocked Intersection 5 76.86 100.00 Yield to Emergency Vehicle 5 70.00 76.00 Mean (route-weighted) 83.87 91.91 Table 4: Performance comparison across long-tail driving scenarios on Bench2Drive-LongTail. SteerVLA demonstrates strong long-tail performance across various scenarios in the C.2 Bench2Drive-LongTail To evaluate the long-tail performance of SteerVLA, we introduce a long-tail subset of Bench2Drive. We focus on 11 categories: 1. Crossing vehicle runs red. A vehicle moving perpendicular to the ego-vehicle runs a red light. The ego-vehicle must recognize that it should wait for the vehicle before it proceeds. 2. Yield to emergency vehicle. An emergency vehicle is driving down the street. The ego-vehicle must yield and wait for the emergency vehicle to pass. 3. Traffic accident. A traffic accident has occurred, and the ego-vehicle must avoid the scene while interacting safely with other vehicles. 4. Roadside cyclist. A cyclist it traveling along the same road as the ego-vehicle. The ego-vehicle must safely avoid the cyclist. 5. Adjacent door opening. A vehicle on the side of the road opens its door into traffic. The ego-vehicle must safely navigate out of the situation while interacting safely with other agents. 6. Jaywalking pedestrian. A pedestrian crosses the street at a non-designated crossing point. The ego-vehicle must slow to wait for the pedestrian to pass. 7. Construction zone. A construction zone has modified the flow of traffic. The ego-vehicle must avoid the construction zone and merge back into the normal traffic flow. 8. Hard brake. A sudden obstacle in the road causes the vehicle to need to brake abruptly. 9. Illegally parked vehicle. A vehicle is parked illegally, obstructing the roadway. The ego-vehicle must reason that the vehicle is in fact parked and navigate around it. 10. Control loss. The ego-vehicle encounters an area of poor traction and loses control. It must recover control without collision. 11. Blocked intersection. Traffic blocks an intersection. The ego-vehicle must reason about the best course of action. Method Traj L2 (m) \downarrow 1s 2s 3s Mean TOKEN ( 45 ) 0.26 0.70 1.46 0.81 PARA-Drive ( 48 ) 0.26 0.59 1.12 0.66 DiMA-VAD ( 18 ) 0.18 0.48 1.01 0.56 GPT-Driver ( 30 ) 0.20 0.40 0.70 0.44 Agent-Driver ( 31 ) 0.16 0.34 0.61 0.37 SteerVLA (Ours) 0.18 0.39 0.63 0.40 Table 5: Open-loop comparison of SteerVLA on the NuScenes planning benchmark. SteerVLA achieves comparable L2 error compared to state-of-the-art methods. The full list of driving scenarios in Bench2Drive is available here. We select what we believe to be the long-tail subset from the list ( 25 ) . We compute the route-weighted mean as Mean=\frac{\sum((\#\text{routes per category})\cdot(\text{DS per category})}{\text{\# routes total}} , where DS is driving score. C.3 Raw values of Performance on Bench2Drive In addition to Fig. 3 and Fig. 4 , we provide the raw scores of the baselines and SteerVLA evaluated on Bench2Drive and Bench2Drive-LongTail, provided in Table 3 and Table 4 . C.4 Failure Cases in Closed-Loop Evaluation However, we still observe failure cases for SteerVLA, which primarily fail into two categories. First, some failures arise from the use of a single front-view camera, which limits visibility of vehicles approaching from the sides or rear (e.g. when yielding to an emergency vehicle). Incorporating multi-view camera inputs is a promising direction for future work. Second, SteerVLA exhibits limited recovery behavior once it enters out-of-distribution states following an incorrect action. For example, when an early or aggressive lane change places the vehicle in an unexpected position relative to surrounding traffic, the policy may fail to recover safely. This limitation is likely due to insufficient coverage of non-optimal behaviors in the training data, which predominantly consists of expert demonstrations. In future work, we plan to address this issue by incorporating co-training or additional supervision from real-world data, where state distributions are more diverse and include recovery behaviors. C.5 Open-Loop Evaluation on Real-World Data We additionally evaluate SteerVLA in an open-loop setting on the NuScenes planning benchmark ( 7 ) to assess performance on real-world driving data. We adopt stronger VLM backbones than in our simulation experiments: Gemma-3 4B ( 44 ) as the high-level policy and PaliGemma ( 3 ) as the low-level policy. For the low-level policy, we follow the approach of 26 , repurposing rarely used tokens to represent discretized actions, where each dimension is divided into 512 uniform bins over the normalized range [-1,1] based on dataset statistics ( 34 ; 6 ) . Since NuScenes does not provide language annotations, we apply our automatic labeling pipeline to generate grounded meta-actions and reasoning traces directly from raw driving data, which are used to supervise both the high-level and low-level policies. Details on data labeling are provided in Sections B.2 and B.3 . All models are trained and evaluated on the official NuScenes training and validation splits. We apply the full auto-labeling pipeline to the NuScenes dataset and evaluate SteerVLA against several baselines on the NuScenes planning benchmark (see Table 5 ). The policy is executed at 2 Hz, and performance is measured using L2 trajectory error over prediction horizons of 1, 2, and 3 seconds. We compare SteerVLA with PARA-Drive ( 48 ) , TOKEN ( 45 ) , DiMA-VAD ( 18 ) , and Agent-Driver ( 31 ) , which represent a range of modular, token-based, distilled, and LLM-guided driving approaches. As shown in Table 5 , SteerVLA achieves performance comparable to or better than existing methods across all horizons, indicating that our framework generalizes effectively to real-world driving data despite being primarily evaluated in closed-loop simulation.