ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Models
Amirhosein Chahe Lifeng Zhou · 2025
Read ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Models by Amirhosein Chahe Lifeng Zhou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10757.txt.
Introduction
Amirhosein Chahe and Lifeng Zhou teach small vision-language models to show their reasoning before answering driving questions, in a 2025 paper. Big models can answer questions about road scenes but rarely explain themselves, a problem for safety-critical cars. The authors used a large model to write structured reasoning chains for scenes from a standard driving benchmark. They then fine-tuned several small model families in three ways: with reasoning, with answers only, and with plain instructions.
Reasoning-based training won consistently, with the best model coming from the Llama 3.2 family. Accuracy and answer quality both improved. The result suggests that compact models meant for cars should learn the habit of explaining, not merely the habit of answering, since their reasoning can be audited after the fact.
Catalog record
| CREATOR | Amirhosein Chahe Lifeng Zhou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10757
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ReasonDrive: Efficient Visual Question Answering for Autonomous Vehicles with Reasoning-Enhanced Small Vision-Language Models Amirhosein Chahe Lifeng Zhou Abstract Vision-language models (VLMs) show promise for autonomous driving but often lack transparent reasoning capabilities that are critical for safety. We investigate whether explicitly modeling reasoning during fine-tuning enhances VLM performance on driving decision tasks. Using GPT-4o, we generate structured reasoning chains for driving scenarios from the DriveLM benchmark with category-specific prompting strategies. We compare reasoning-based fine-tuning, answer-only fine-tuning, and baseline instruction-tuned models across multiple small VLM families (Llama 3.2, Llava 1.5, and Qwen 2.5VL). Our results demonstrate that reasoning-based fine-tuning consistently outperforms alternatives, with Llama3.2-11B-reason achieving the highest performance. Models fine-tuned with reasoning show substantial improvements in accuracy and text generation quality, suggesting explicit reasoning enhances internal representations for driving decisions. These findings highlight the importance of transparent decision processes in safety-critical domains and offer a promising direction for developing more interpretable autonomous driving systems. The code and dataset is available at https://github.com/Zhourobotics/ReasonDrive . Perception Question: What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision. Thinking: Observe the scene and note a busy urban environment and construction on the sides. Identify key objects, focusing on the positions relative to the ego vehicle: a black car and a white sedan ahead, a silver SUV behind, and a person in a wheelchair to the front right. Note visual cues such as the pedestrian sign indicating yielding to pedestrians in front of the ego vehicle. Input the identified objects and their positions as crucial for future driving decisions. Answer: There is a black car to the front of the ego vehicle, a white sedan to the front of the ego vehicle, a silver SUV to the back of the ego vehicle, a person in a wheelchair to the front right of the ego vehicle, and a yield to pedestrians to the front of the ego vehicle. The IDs of these objects are <c1> , <c2> , <c3> , <c4> , and <c5> . Prediction Question: Is <c1> a traffic sign or a road barrier? Thinking: Analyze the location and characteristics of the object in question based on its coordinates <c1> relative to the camera. Determine if the object possesses common features of a traffic sign, such as specific shapes, colors, or placement above ground level. Check if the object resembles typical road barriers, which are usually positioned at ground level, often elongated, and can be movable or fixed. Given the data input, evaluate whether it matches the expected attributes of either a traffic sign or a road barrier. The identification process shows no match for typical features of traffic signs or road barriers; thus, conclude the object is neither. Answer: No. Planning Question: In this scenario, what are safe actions to take for the ego vehicle? Thinking: The scenario involves navigating a street partially obstructed by construction. Safety assessment dictates prioritizing the avoidance of sudden braking to maintain stability and control, especially around construction areas. Slightly offsetting to the left reduces the risk of collision with obstacles on the right without veering into oncoming traffic, optimizing lane position. Decelerating gradually ensures the vehicle adapts to the changing environment smoothly, minimizing the risk of losing control or causing discomfort to passengers. This approach balances maintaining safety and ensuring efficient movement through the construction zone. Answer: Slightly offset to the left, decelerate gradually without braking. Slightly offset to the left, decelerate gradually without braking. Behavior Question: Predict the behavior of the ego vehicle. Thinking: 1) Identify the road structure: The images show a straight road with lane markings directing vehicles to continue straight. 2) Consider environmental factors: Construction cones and pedestrian signs indicate a need for caution, likely leading to slower speeds. 3) Analyze motion pattern and trajectory: Observing the surrounding vehicles and lane markings implies the ego vehicle is maintaining its straight trajectory. 4) Determine intent based on context: Due to construction and pedestrian considerations, the ego vehicle is likely driving slowly to ensure safety. Answer: The ego vehicle is going straight. The ego vehicle is driving slowly. Figure 1 : Example frame from our reasoning-enhanced dataset derived from DriveLM. Each frame consists of six camera views (top) and four categories of driving tasks (bottom) with the original question, generated reasoning, and final answer. (a) Front Left (b) Front (c) Front Right (d) Back Left (e) Back (f) Back Right 1 Introduction Visual Question Answering (VQA) represents a critical advancement in autonomous driving technology, enabling vehicles to understand and respond to natural language questions about their visual surroundings. This capability bridges the gap between computer vision and natural language processing, allowing autonomous vehicles to interpret road scenes and communicate their understanding in human-readable formats Rekanar et al. 2023 ; Chahe and Zhou 2025 . As a safety-critical application, autonomous driving necessitates explainable decision-making, which VQA can facilitate through question-answering-based causal reasoning Atakishiyev et al. 2023 . Applying VQA to autonomous driving presents unique challenges, including processing multi-modal data, analyzing multi-frame sequences from continuous real-time acquisition, and interpreting complex outdoor scenes containing both moving and static elements Qian et al. 2023 . To address these complexities, researchers have developed specialized datasets and benchmarks, such as NuScenes-QA, which encompasses 34,000 visual scenes and 460,000 question-answer pairs specifically designed for autonomous driving scenarios Qian et al. 2023 . The integration of VQA in autonomous vehicles serves multiple purposes beyond basic perception. It enables natural interaction between the vehicle and its occupants, fosters trust in autonomous technology, and makes the vehicle’s decision-making process more transparent Rekanar et al. 2024 . This transparency is crucial for building user confidence in autonomous systems, as it allows vehicles to communicate their actions, intentions, and reasoning in human-comprehensible language. Recent large vision-language models (VLMs) Hurst et al. 2024 ; Team et al. 2024 ; Anthropic 2024 offer promising alternatives to traditional domain-specific approaches, potentially performing multiple perception tasks within a unified framework Dewangan et al. 2023 . However, deploying these large models in vehicles presents significant challenges due to computational constraints and the need for interpretable reasoning. While benchmarks like DriveLM Sima et al. 2023 have advanced VLM applications in driving scenarios, a critical gap remains in making these capabilities accessible within real-world computational constraints without sacrificing reasoning transparency. Current approaches often rely on massive foundation models impractical for vehicle deployment or compromise on transparency for efficiency Cui et al. 2024 . We address this gap by investigating whether explicitly modeling reasoning during fine-tuning enhances smaller, deployable VLMs on driving decision tasks. Our contributions include: 1. A reasoning-enhanced dataset derived from DriveLM using GPT-4o to generate structured reasoning chains for driving scenarios. 2. Comprehensive evaluation of reasoning-based fine-tuning across multiple small VLM families (Llama 3.2, Llava 1.5, and Qwen 2.5VL), demonstrating that reasoning-based approaches consistently outperform alternatives. 3. Evidence that explicit reasoning enhances internal representations for driving decisions, with Llama3.2-11B-reason achieving the highest overall performance while maintaining interpretability. Our work shows that reasoning-enhanced fine-tuning creates efficient, interpretable models that address both computational and safety requirements for autonomous vehicles. Figure 1 presents the structure of reasoning and corresponding answers for an example frame from the DriveLM dataset. 2 Related Work 2.1 VQA and