Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidance

Fengze Yang, Bo Yu, Yang Zhou, Xuewen Luo, Zhengzhong Tu, Chenxi Liu · 2025

Read Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidance by Fengze Yang, Bo Yu, Yang Zhou, Xuewen Luo, Zhengzhong Tu, Chenxi Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01057.txt.

Introduction

Fengze Yang and co-authors build an edge system that fuses road sensor data with vision-language models to dodge crashes. Cars that sense only onboard miss far or hidden hazards. Roadside data can warn them earlier, if fused fast enough.

Their planner joins infrastructure alerts with the car's own sensors. Visual codes capture traffic motion and driver intent nearby. Symbolic inputs carry exact numbers the model must respect. A tuned light vision-language model reasons over both streams. It then shapes a safer path for the car. Residual trajectory ideas support robust real-time running. Edge hardware limits shaped the whole design. The work targets split-second safety on busy roads.

Catalog record

CREATORFengze Yang, Bo Yu, Yang Zhou, Xuewen Luo, Zhengzhong Tu, Chenxi Liu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE75 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01057

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidance Fengze Yang, Bo Yu, Yang Zhou, Xuewen Luo, Zhengzhong Tu, Chenxi Liu Abstract Autonomous driving (AD) systems relying solely on onboard sensors may fail to detect distant or obstacle hazards, potentially causing preventable collisions; however, existing transformer-based Vehicle-to-Everything (V2X) approaches, which mitigate AD sensing limitations, either lack effective multimodal fusion and reasoning or struggle to meet real-time performance requirements under complex, high-dimensional traffic conditions. This paper proposes the R eal-time E dge-based A utonomous C o-pilot T rajectory planner (REACT), a V2X-integrated trajectory optimization framework for AD based on a fine-tuned lightweight Vision-Language Model (VLM). REACT integrates infrastructure-provided hazard alerts with onboard sensor data, capturing intricate surrounding traffic dynamics and vehicle intents through visual embeddings, interpreting precise numerical data from symbolic inputs, and employing contextual reasoning to generate optimized, safety-oriented trajectories. To ensure robust real-time deployment on edge devices, REACT innovatively employs Residual Trajectory Fusion (RTF) design and specialized edge-adaptation strategies to reduce model complexity and improve inference efficiency. Evaluated on the DeepAccident benchmark, REACT achieves state-of-the-art performance, a 77% collision rate reduction, a 48.2% Video Panoptic Quality (VPQ), and a 0.57-second inference latency on the Jetson AGX Orin. Ablation studies validate the contribution of each input, module, and edge adaptation strategy. These results highlight the effectiveness of lightweight VLMs in enabling real-time cooperative planning on edge platforms and underscore the potential of language-guided contextual reasoning for improving traffic safety and responsiveness. Keywords Large Language Model \cdot Multi-modal Data Fusion \cdot Spatial-temporal Alignment \cdot Transportation Safety \cdot Connected and Autonomous Vehicles \cdot Edge Computing 1 Introduction Road traffic crashes remain a critical global issue, claiming approximately 1.19 million lives and injuring up to 50 million individuals annually. They are the leading cause of death among individuals aged 5 to 29 [ 1 ] . Among the contributing factors, impaired, distracted, or careless driving stands out as the primary cause, accounting for 29.2% of all accidents and resulting in over 3,000 fatalities each year in the U.S. alone [ 2 , 3 ] . This significant toll underscores the urgent need for autonomous driving (AD) and Advanced Driver-Assistance Systems (ADAS) to help mitigate human error and reduce distraction-related collisions [ 4 ] . Although these technologies hold promise for enhancing road safety, their capabilities are often limited by the perception range of onboard sensors, which can delay or prevent the detection of occluded or distant hazards. As a result, ADAS and AD systems may fail to respond effectively to unexpected threats, highlighting the necessity for a more proactive and context-aware driving framework [ 5 , 6 ] . To overcome the safety limitations of isolated onboard sensing, connected autonomous vehicle (CAV) networks leverage Vehicle-to-Everything (V2X) communication to integrate diverse transportation agents, including vehicles, roadside units (RSUs), and traffic management entities. This collaborative framework enhances system-wide situational awareness, enabling individual vehicles to perceive beyond their immediate sensor range, including blind spots and broader traffic conditions, thereby supporting more proactive and informed decision-making [ 7 ] . Despite its potential, large-scale V2X deployment faces notable challenges in prevailing perception methodologies. Models such as Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Vision Transformers (ViTs) [ 8 ] often suffer from limited generalization, requiring extensive labeled datasets and struggling with novel or unseen scenarios [ 9 , 10 ] . Moreover, non-attention-based methods excel primarily in pattern recognition but fail to meet the high-level contextual reasoning required for complex real-world driving [ 11 , 12 ] . Besides the aforementioned limits, integrating multimodal information remains a substantial challenge due to the architectural complexity and limitations of existing model designs [ 13 , 14 ] . Recent advances in Vision-Language Models (VLMs), extensions of Large Language Models (LLMs) with integrated visual processing capabilities [ 15 ] , present promising solutions to overcome the limitations of conventional perception systems [ 16 , 17 ] . Modern traffic environments are dynamic, complex, and context-rich [ 18 ] , demanding autonomous systems that can understand and respond to diverse multimodal inputs. VLMs address this challenge by combining semantic reasoning with visual context, enabling more informed and adaptive decision-making, outperforming conventional models [ 16 ] . First, VLMs excel in high-level contextual reasoning, interpreting spatiotemporal cues, anticipating agent intent, and understanding implicit traffic rules, which are essential for navigating scenarios like occluded merges, crowded intersections, or rare accident scenes [ 19 , 20 ] . Second, VLMs inherently support multimodal integration, effectively fusing heterogeneous sensor data and textual inputs [ 21 ] , which facilitates comprehensive perception and enables more accurate, adaptive, and timely decision-making in real-world driving scenarios [ 22 , 23 ] . Third, VLMs demonstrate strong generalization through zero-shot learning [ 24 , 25 ] , reducing the dependency on large-scale labeled datasets and enhancing the robustness and reliability of perception systems. Finally, their ability to pair linguistic explanations with visual perception improves model interpretability [ 26 , 27 ] , supporting more efficient human-vehicle interactions by winning the trust of human drivers. Building upon these insights, this research introduces a Real-time Edge-based Autonomous Co-pilot Trajectory planner (REACT), a V2X-enabled framework leveraging a lightweight VLM to improve transportation safety. REACT comprises five core modules: 1) sensing data acquisition, 2) data structuring, 3) task alignment, 4) task projection enhancement, and 5) Residual Trajectory Fusion (RTF). The sensing module gathers multimodal inputs, including onboard camera frames, navigation goals, ego vehicle states, and hazard alerts from Roadside Units (RSUs). Visual data is converted into Bird’s-Eye View (BEV) maps that capture road geometry and nearby road users. To support VLM’s motion reasoning, REACT stacks BEV maps from the current moment and a prior timestamp, allowing the VLM to infer agent trajectories and intentions. The data structuring module filters noise, extracts necessary information, and validates the RSU hazards. The task alignment stage aligns input coordination and timestamps, then encodes both visual and symbolic information into optimized prompts. The task projection enhancement module then passes those prompts into a lightweight VLM, which, guided by Chain-of-Thought prompting, predicts necessary trajectory corrections. Finally, the RTF module applies these residuals to the nominal trajectory, yielding a refined path that proactively avoids potential collisions. A notable feature of REACT is its edge-based deployment, which ensures real-time responsiveness and minimizes delays caused by network latency and bandwidth limitations with edge computing [ 28 , 29 ] . VLMs demand substantial computational resources, often exceeding the capabilities of edge devices [ 28 ] . To address this challenge, REACT introduces innovative optimization strategies that combine prompt engineering [ 30 ] with fine-tuning techniques on the lightweight model [ 31 , 32 ] , achieving a model reaction latency reduction of 93.67% without sacrificing performance. In terms of model architecture, REACT incorporates a novel module, RTF, which leverages original trajectory data to enhance the outputs of the VLM. The RTF module is compact, modular, and readily adaptable for deployment in other trajectory prediction tasks. Experimental results indicate that the combination of REACT’s optimization strategies and the RTF module improves system efficiency by 95.25%, enhances trajectory prediction accuracy, and ensures greater training stability. In summary, REACT addresses key limitations of conventional perception methodologies

Read the complete text