HMVLM: Multistage Reasoning-Enhanced Vision-Language Model for Long-Tailed Driving Scenarios

Daming Wang, Yuhao SongZijian HeKangliang ChenXing PanLu Deng, Weihao GuHAOMO.AI [email protected] [email protected] · 2025

Read HMVLM: Multistage Reasoning-Enhanced Vision-Language Model for Long-Tailed Driving Scenarios by Daming Wang, Yuhao SongZijian HeKangliang ChenXing PanLu Deng, Weihao GuHAOMO.AI [email protected] [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05883.txt.

Introduction

Daming Wang and co-authors presented HMVLM in 2025, a vision-language model that plans driving the slow, thoughtful way. Self-driving stacks pair a fast reflex controller with a slower planner. Their planner reads five camera views plus four seconds of the car's own motion history. It then reasons in stages: understand the scene, decide the move, infer the path.

That staged chain of thought targets the rare, awkward cases that dominate real driving risk. A spline smoothing step irons out late jitters and sharp kinks in the planned path. Trained on the Waymo Open Dataset, the system scored near the top of a 2025 end-to-end driving challenge. It placed second and beat the public baseline by a clear margin. The authors frame the design as cognition-inspired: fast hands, slow head.

Catalog record

CREATORDaming Wang, Yuhao SongZijian HeKangliang ChenXing PanLu Deng, Weihao GuHAOMO.AI [email protected] [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE13 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05883

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HMVLM: Multistage Reasoning-Enhanced Vision-Language Model for Long-Tailed Driving Scenarios Daming Wang, Yuhao SongZijian HeKangliang ChenXing PanLu Deng, Weihao GuHAOMO.AI [email protected] [email protected] Abstract We present HaoMo Vision–Language Model (HMVLM) , an end-to-end driving framework that implements the slow branch of a cognitively inspired fast–slow architecture. A fast controller outputs low-level steering, throttle, and brake commands, while a slow planner-a large vision-language model-generates high-level intents such as ”yield to pedestrian” or ”merge after the truck” without compromising latency. HMVLM introduces three upgrades: (1) selective five-view prompting with an embedded 4 ​ s 4\,\mathrm{s} history of ego kinematics, (2) multi‑stage chain‑of‑thought (CoT) prompting that enforces a Scene Understanding → \rightarrow Driving Decision → \rightarrow Trajectory Inference reasoning flow, and (3) spline-based trajectory post-processing that removes late-stage jitter and sharp turns. Trained on the Waymo Open Dataset, these upgrades enable HMVLM to achieve a Rater Feedback Score(RFS) of 7.7367 , securing 2 nd place in the 2025 Waymo Vision-based End-to-End(E2E) Driving Challenge and surpassing the public baseline by 2.77% . 1 Introduction Modular pipelines inflate architectural complexity, incur cumulative information loss at every interface, and are difficult to optimise globally because each module pursues its own objective. In contrast, an End-to-End (E2E) approach makes the entire chain differentiable, allowing direct optimisation for the final control task, collapsing hand-tuned interfaces, sharing one backbone for efficiency, and leveraging data-driven scaling that can unlock emergent abilities. The 2025 Waymo Open Dataset Vision-based End-to-End Driving Challenge curates 4,021 20\,\mathrm{s} driving segments that focus on long-tail hazards—construction detours during marathons, pedestrians falling from scooters, debris on freeways—events that occur in fewer than <0.003\% of daily driving. This setting provides a rigorous arena for evaluating the robustness and generalisation of E2E methods on rare yet safety-critical scenarios. Purely fast controllers such as VAD ( Jiang et al. 2023 ) compress the scene into vector tokens and refresh their plan with millisecond latency, but their short temporal horizon limits commonsense reasoning. At the opposite extreme, slow-only designs like EMMA ( Hwang et al. 2024 ) recast perception, prediction, and planning as a single multimodal VLM problem, reaching state-of-the-art motion-planning accuracy at the cost of heavy compute and inference delays. The field is therefore converging on dual fast–slow pipelines: DriveVLM-Dual ( Tian et al. 2024 ) attaches a language-based planner to a real-time stack, while Senna pairs a Senna ( Jiang et al. 2024 ) planner with an end-to-end regressor, simultaneously harvesting VLM commonsense and deterministic control. Following this trend, we also adopt a dual-system architecture. In this competition submission we expose only the slow system, a low-frequency, VLM-powered decision module responsible for semantic understanding and strategic reasoning in complex scenarios. HMVLM serves as the VLM–driven slow planner in our fast–slow architecture. Trained solely on the Waymo Open Dataset, it attains a Rater Feedback Score of 7.7367 , securing 2 nd place in the 2025 Vision-based End-to-End Driving Challenge and outperforming the public baseline by 2.77% . Our work delivers three key contributions: 1) Selective multi-view prompting with kinematic context . Five surround-view images, together with a 4\,\mathrm{s} history of ego velocity and acceleration, are injected directly into the prompt, reducing input bandwidth while improving motion-prediction fidelity. 2) Multi‑stage chain‑of‑thought (CoT) prompting . Special tokens enforce a three-stage reasoning flow. Scene Understanding \rightarrow Driving Decision \rightarrow Trajectory Inference, providing interpretable intermediate text that boosts human-rater trust. 3) Spline-based trajectory smoothing . Post-processing removes late-stage oscillations and sharp kinks, markedly reducing collision events. Figure 1 : Overall diagram of HMVLM. 1 1 footnotetext: Corresponding authors. 2 Methodology HMVLM is an end-to-end autonomous driving framework built upon a general-purpose vision-language model. The proposed method progressively reasons from scene understanding to driving decisions and trajectory inference by leveraging multi-stage Chain-of-Thought (CoT) prompting Wei et al. 2022 and trajectory refinement. Figure 1 illustrates the overall pipeline. 2.1 Network Design and data pipeline HMVLM takes multi-view images as input. To balance computational efficiency and driving performance, we select three front-facing views along with left and right side views. The output is a structured, multi-stage reasoning result that includes scene understanding, driving decision-making, and trajectory prediction. For baseline model selection, we conducted a systematic comparison and evaluation of several open-source vision-language models in terms of visual perception, hallucination tendencies Liu et al. 2024 , and reasoning capabilities under general scenarios Duan et al. 2024 . These dimensions are critical for autonomous driving, where safety and reliability are paramount. The overall architecture is customized and extended from the general-purpose foundation model Qwen2.5-VL-3B ( Bai et al. 2025 ) . Multimodal image-text pairs serves as the core training fuel for HMVLM. This work leverages a pre-trained large language model to perform a large-scale automated annotation on the Waymo Open Dataset, constructing two key VQA tasks aligned with the reasoning stages of HMVLM: scene understanding and driving decision making. For scene understanding, prompt engineering is used to guide model attention toward driving-relevant elements. The model is instructed to output structured natural language descriptions that follow a specific tone and style. For the driving decision task, we incorporate ground truth trajectory data, navigation instructions, and the historical state of the ego vehicle into the model input. This helps the labeling model infer realistic human-like driving intentions and generate high-quality visual-text-action aligned annotations, which are crucial for downstream tuning and structured reasoning in HMVLM. 2.2 Multi-stage Reasoning In the large model research community, it has been repeatedly demonstrated that Chain-of-Thought (CoT) reasoning significantly boosts the quality of model outputs Xu et al. 2024 . HMVLM leverages this insight to enhance planning performance in autonomous driving. It decomposes the answer generation process into three structured reasoning stages: • Scene Understanding . In the initial stage, HMVLM provides a global summary of the current multi-view visual input, with a particular focus on driving-relevant factors such as salient obstacles, traffic lights, lane markings, weather, and visibility. • Driving Decision . Building upon the visual understanding, the model infers driving intentions based on the vehicle’s historical states and navigation goals, expressed in natural language to describe upcoming maneuvers, such as “prepare to turn left and slow down to yield”. • Trajectory Inference . Finally, the model translates the preceding reasoning into a structured output, a sequence of future trajectory points in the bird’s-eye view (BEV) coordinate space. Specifically, enabling CoT-style reasoning in HMVLM requires careful design of the data flow. Instead of relying on lengthy multi-turn dialogue-style inference, HMVLM introduces dedicated special tokens to structurally define its three-stage reasoning process Jiang et al. 2025 . This not only reduces redundancy but also enhances the model’s ability to interpret and generate semantically aligned outputs. The expected output of the model follows a structured format: <DESC_START> “textual description of scene understanding” <DESC_END><DECI_START> “natural language driving decision” <DECI_END><TRAJ_START> “predicted trajectory waypoints” <TRAJ_END> . This structured reasoning paradigm enhances both the interpretability and robustness of HMVLM’s decision-making. By explicitly disentangling visual understanding, intention, and motion generation, the model is better equipped to generalize across diverse driving scenarios. 2.3 Trajectory Refinement To support HMVLM’s structured reasoning and enhance the reliability of predicted trajectories, trajectory refinement is employed to address potential kinematic discontinuities and irregularities in trajectory outputs. A smoother trajectory reduces unnecessary vehicle maneuvers and enhances safety and passenger comfort. We implement

Read the complete text