Knowledge Graphs as World Models for Semantic Material-Aware Obstacle Handling in Autonomous Vehicles
Ayush Bheemaiah & Seungyong Yang · 2025
Read Knowledge Graphs as World Models for Semantic Material-Aware Obstacle Handling in Autonomous Vehicles by Ayush Bheemaiah & Seungyong Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21232.txt.
Introduction
Ayush Bheemaiah and Seungyong Yang ask if a self-driving car can judge what a road block is made of. Their paper is from two thousand twenty-five. Cars sense blocks with cameras and laser scans, but those tools say little about traits such as softness, weight, or stretch. Those traits shape whether a car should swerve, stop, or drive on.
They add a knowledge graph as a world model, which links each block to facts about its stuff. In the CARLA test world, the graph-equipped car dealt with blocks better than a plain car. It changed lanes for hard blocks and drove over soft bags. It also stopped more often when sensors clashed.
Catalog record
| CREATOR | Ayush Bheemaiah & Seungyong Yang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-21232
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Knowledge Graphs as World Models for Semantic Material-Aware Obstacle Handling in Autonomous Vehicles Ayush Bheemaiah & Seungyong Yang Abstract The inability of autonomous vehicles (AVs) to infer the material properties of obstacles limits their decision-making capacity. While AVs rely on sensor systems such as cameras, LiDAR, and radar to detect obstacles, this study suggests combining sensors with a knowledge graph (KG)-based world model to improve AVs’ comprehension of physical material qualities. Beyond sensor data, AVs can infer qualities such as malleability, density, and elasticity using a semantic KG that depicts the relationships between obstacles and their attributes. Using the CARLA autonomous driving simulator, we evaluated AV performance with and without KG integration. The findings demonstrate that the KG-based method improves obstacle management, which allows AVs to use material qualities to make better decisions about when to change lanes or apply emergency braking. For example, the KG-integrated AV changed lanes for hard impediments like traffic cones and successfully avoided collisions with flexible items such as plastic bags by passing over them. Compared to the control system, the KG framework demonstrated improved responsiveness to obstacles by resolving conflicting sensor data, causing emergency stops for 13.3% more cases. In addition, our method exhibits a 6.6% higher success rate in lane-changing maneuvers in experimental scenarios, particularly for larger, high-impact obstacles. While we focus particularly on autonomous driving, our work demonstrates the potential of KG-based world models to improve decision-making in embodied AI systems and scale to other domains, including robotics, healthcare, and environmental simulation. 1 Introduction Autonomous vehicles (AV) have gained rapid market penetration in recent years, yet the industry’s goal of achieving Level 5 automation remains extremely difficult, with forecasts suggesting that fully self-driving cars will not be available until 2035 ( S&P Global Mobility 2023 ) . The transition from current AVs at Level 3, which still require human supervision, to complete autonomy under any Operational Design Domain (ODD) remains distant ( SAE International 2021 ) . Therefore, fundamental approaches to autonomous vehicle development must be reevaluated. The current AVs available to the public highly depend on visual perception that mirrors human visual cognition through heterogeneous sensors such as cameras, LiDAR, and RADAR ( Author & Researcher 2021 ) . However, this approach oversimplifies the complex decision-making processes involved in human driving, as human drivers integrate multiple layers of knowledge that transcend beyond visual and spatial recognition using sensors: (i) recognition of physical properties of obstacles, (ii) moral and emotional considerations, (iii) anticipatory behavior in driving (iv) contextual and situational understanding ( Scientist & Engineer 2020 ) . AVs primarily show low performance in performing contextual decisions about common scenarios related to obstacle management, such as distinguishing the appropriate response to a lightweight plastic bag placed on the road ( Developer & Creator 2020 ) . Unlike human drivers, AVs fail to replicate the intuitive understanding required to navigate obstacles and fall short in understanding physical properties to optimize their decision-making and minimize the damage after a potential collision with an object. More specifically, lightweight objects like empty plastic bags can be safely driven over, while rigid paper boxes require lane changes. Hence, while human drivers can assess such obstacles by instinctively understanding physical properties, AVs cannot incorporate material characteristics into their reasoning ( Goodfellow et al. 2016 ) . Thus, a comprehensive framework that amalgamates the recognition of material properties with existing heterogeneous sensor systems is fundamental for enabling AVs to handle obstacles that exceed the interpretative capabilities of existing sensors. Human drivers, when faced with atypical situations such as a large plastic bag blowing across the road, rely on a combination of visual perception, situational judgment, and adaptive behavior to make their decisions ( Technologist & Innovator 2024 ) . By accurately assessing the object’s size, shape, texture, and movement patterns visually and intuitively, they evaluate its potential threat, allowing them to make quick decisions, such as whether to avoid the object or continue driving without alteration. Whereas, AVs rely on pre-trained algorithms and sensor data, which may not cover all possible scenarios ( Researcher & Scholar 2024 ) . Therefore, this study aims to enhance the autonomous vehicle’s ability to infer and respond to various obstacles it has never encountered before by implementing a KG-based framework within the CARLA autonomous driving simulator through an application programming interface (API) integration ( Dosovitskiy et al. 2017 ) . By utilizing graph databases, which excel at representing and processing complex relationships between objects and their physical properties ( Anuyah et al. 2024 ) , our work successfully integrates a KG approach on the sensor-based perception within the CARLA simulator. 2 Related Work 2.1 Overall Sensors Summary First, we examine the function of the three major sensors (cameras, LiDAR, and radars) used in AVs. (i) Cameras produce images of the surrounding environment by detecting light reflected or emitted from objects onto a photosensitive surface. ( Investigator & Analyst 2020 ) . Additionally, both moving and stationary obstacles are identifiable by cameras with high-resolution photographs. (ii) LiDAR (Light Detection And Ranging) is a distance-sensing technique that emits laser light pulses or infrared beams toward target impediments. By calculating the time between emission and reception of the light pulse, LiDAR obtains an accurate distance estimate ( Expert & Specialist 2020 ) . LiDAR produces data in the form of point cloud data (PCD), which consists of the (x,y,z) coordinates and the intensity information of the obstacles ( Professional & Consultant 2023 ) . (iii) Radars determine the relative speed and position of target obstacles using the Doppler shift in electromagnetic waves ( S&P Global Mobility 2023 ) . As the target approaches the radar, the frequency of the detected signal increases, enabling AVs to determine range information ( Pioneer & Visionary 2013 ) . 2.2 Problems Associated with Sensors Cameras, LiDAR, and radars each have weaknesses when operating independently and also collectively. Cameras show high performance for capturing high-resolution images but struggle to function in poor climatic conditions such as fog or heavy rain and are unable to reliably measure distance ( Scientist & Researcher 2020 ) . LiDAR is prone to data distortions such as "black holes" in its point cloud and is not suited for poor weather. Radar performs well in detecting obstacles at long distances and in poor climatic conditions but it is unreliable in distinguishing between complex materials or those with weak echoes, such as plastic or rubber ( Zhang & Wang 2021 ) . The issues of unreliability and inaccuracy are amplified when these sensors are combined, a process referred to as sensor fusion, where data from multiple sources are merged for perception ( Yamamoto & Tanaka 2014 ) . Current sensor fusion algorithms rely on object-level fusion that involves sensors identifying and classifying objects independently before combining their data. This approach is not optimized as it fails to integrate the strengths of each sensor into a unified collective understanding of the environment ( Scientist & Engineer 2020 ) . Furthermore, inconsistencies between sensor inputs can lead to conflicting interpretations which reduces the reliability of AVs. For instance, a lightweight plastic bag might be misclassified as a rigid obstacle due to conflicting data from LiDAR and radar. 2.3 Convolutional Neural