Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning
Shengyuan Wang, Jie Feng, Tianhui Liu, Dan Pei, Yong Li · 2025
Read Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning by Shengyuan Wang, Jie Feng, Tianhui Liu, Dan Pei, Yong Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-19586.txt.
Introduction
Shengyuan Wang, Jie Feng, Tianhui Liu, Dan Pei, and Yong Li measure and mend language models' false geography, in a 2025 paper. Models answer map questions fluently, yet they invent places, misplace cities, and contradict themselves — geospatial hallucinations that poison uses like travel planning and local forecasting.
The team build an evaluation frame on structured knowledge graphs, where facts are controlled, and grade twenty advanced models. They then propose a training method that aligns the model's factuality dynamically, during use rather than only in advance. Scores from the benchmark show how far even top models stray on place knowledge. A standard yardstick plus a fix gives map-facing AI a path to trustworthiness.
Catalog record
| CREATOR | Shengyuan Wang, Jie Feng, Tianhui Liu, Dan Pei, Yong Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-19586
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning Shengyuan Wang, Jie Feng, Tianhui Liu, Dan Pei, Yong Li Abstract Large language models (LLMs) possess extensive world knowledge, including geospatial knowledge, which has been successfully applied to various geospatial tasks such as mobility prediction and social indicator prediction. However, LLMs often generate inaccurate geospatial knowledge, leading to geospatial hallucinations—incorrect or inconsistent representations of geospatial information—that compromise their reliability. While the phenomenon of general knowledge hallucination in LLMs has been widely studied, the systematic evaluation and mitigation of geospatial hallucinations remain largely unexplored. To address this gap, we propose a comprehensive evaluation framework for geospatial hallucinations, leveraging structured geospatial knowledge graphs for controlled assessment. Through extensive evaluation across 20 advanced LLMs, we uncover the hallucinations in their geospatial knowledge. Building on these insights, we introduce a dynamic factuality aligning method based on Kahneman-Tversky Optimization (KTO) to mitigate geospatial hallucinations in LLMs, leading to a performance improvement of over 29.6% on the proposed benchmark. Extensive experimental results demonstrate the effectiveness of our benchmark and learning algorithm in enhancing the trustworthiness of LLMs in geospatial knowledge and reasoning tasks. Codes and data are available via https://anonymous.4open.science/r/GeospatialHallucination-823A/ . 1 Introduction Figure 1: An overview of our work. In order to address real-world geospatial hallucination of LLMs. We propose 1) GeoHaluBench to detect and evaluate geospatial knowledge errors, 2) DynamicKTO to enhance LLMs’ factuality effectively and efficiently. Recently, large language models (LLMs), known for their excellent reasoning abilities Wei et al. 2022 and extensive world knowledge Yu et al. 2023 ; Ivanova et al. 2024 , have been widely applied across various domains.The extensive geospatial knowledge embedded in LLMs has also been explored Gurnee and Tegmark 2023 ; Roberts et al. 2023 and successfully applied to various geospatial tasks. Gurnee et al. Gurnee and Tegmark 2023 and Roberts et al. Roberts et al. 2023 demonstrate that LLMs maintain grounded geospatial knowledge that accurately reflects the real world. Leveraging this geospatial knowledge, LLM-based methods have shown promising performance in several geospatial tasks, including GeoLLM for social indicator prediction Manvi et al. 2023 , AgentMove for global mobility prediction Feng et al. 2024b , and UrbanCLIP for robust and effective urban representation Yan et al. 2024 . While the world knowledge embedded in LLMs has contributed to their widespread success in various applications over the past two years, researchers have identified significant errors and self-contradictions—referred to as hallucinations—in the generated results of LLMs, particularly in domain-specific areas Huang et al. 2023 ; Ji et al. 2023 . These hallucinations significantly affect the trustworthiness of LLMs and their performance in real-world applications. Geospatial knowledge within LLMs also exhibits notable hallucinations in practical use cases Manvi et al. 2024 ; Feng et al. 2024c . Detecting and mitigating these hallucinations has become a critical problem for the development and reliable deployment of LLMs in the real world. While various solutions have been proposed for general and domain-specific knowledge hallucinations Zhang et al. 2024 ; Chen et al. 2024 , systematic evaluation and mitigation of geospatial knowledge hallucinations remain largely unexplored. This is particularly challenging due to two main factors. First, geospatial data is complex and diverse, resulting in varied manifestations of associated spatial hallucinations. Second, general hallucination reduction methods often fail to account for the unique characteristics of geospatial knowledge, e.g., various elements and relations between them. In this paper, we propose a systematic benchmark and an effective factuality aligning method to evaluate and mitigate geospatial hallucinations in LLMs. We first build SpatialKG to reorganize the diverse and unstructured geospatial data, and introduce a set of geospatial evaluation questions, along with a systematic taxonomy of hallucinations. Furthermore, the elements in the evaluation questions are flexible and customizable, allowing for a thorough localization of the shortcomings in geospatial hallucinations across various LLMs. After evaluating 20 advanced LLMs using our proposed benchmark, we find that most of them, especially the open-source LLMs, exhibit significant hallucinations. Based on our observations of these geospatial hallucinations, we develop DynamicKTO to effectively mitigate the geospatial hallucinations in smaller-scale, open-source LLMs by enhancing the KTO algorithm with dynamic factuality aligning. With the help of DynamicKTO , LLama3.1-8B achieves a significant performance improvement on the proposed benchmark and demonstrates competitive performance compared to the second-best model among the 20 advanced LLMs we evaluated. In summary, our contributions are threefold. • To the best of our knowledge, we are the first to systematically evaluate and mitigate the geospatial hallucinations in LLMs. • We have developed a comprehensive benchmark to assess geospatial hallucinations and analyze the performance of 20 advanced LLMs within this framework. • We propose DynamicKTO , which extends the KTO by incorporating dynamic factuality aligning to account for the diversity and heterogeneity of geospatial knowledge and data. • Extensive experiments on GeoHaluBench and DynamicKTO demonstrate the effectiveness of our proposed framework in evaluating and mitigating geospatial hallucinations. 2 Methods 2.1 SpatialKG : Structured Geospatial Knowledge Organization As a reliable and informative knowledge base is essential for the definition, detection, and mitigation of the hallucination problem. We construct a high-quality knowledge graph called SpatialKG based on previous work Liu et al. 2023b . We design a new schema (the high-level structure of KG, including the types of entities and relations) in order to capture fundamental elements in the urban environment and to cover most important relations for geospatial cognition. In SpatialKG , fundamental entities include Point of Interest (POI), Area of Interest (AOI) and Road as basic elements describing urban and rural structures. Based on the types of entity in SpatialKG , we conclude the typical and important relations to describe the spatial connections between entities as follows: POI-LocateAt-AOI, POI-Near-POI, AOI-Near-AOI, AOI-ConnectTo-Road, and Road-Intersect-Road. Mastering the real-world knowledge does not only imply the memory of existing entities’ names, but also refers to the capability of recognizing their important attributes. We select the following attributes and link them to the entities in SpatialKG . For POI, address and category are basic information considered. For regions, attributes include area and type of land use (industrial, residential, etc.). As for Road, we select the length of a road as its attribute discussed. SpatialKG is automatically constructed from OpenStreetMap 1 1 1 https://www.openstreetmap.org and Foursquare’s Open Source Places 2 2 2 https://opensource.foursquare.com/os-places/ , which are updating and high-quality city data sources. We design a pipeline to examine and filter the original data for quality control. Figure 2: An illustration of the hierarchy and composition of GeoHaluBench . 2.2 GeoHaluBench : Geospatial Hallucination Benchmarking With adequate knowledge from SpatialKG , we can classify and benchmark LLM’s geospatial hallucination of real world. 2.2.1 GeoHaluBench Composition In this section, we introduce the overall composition of GeoHaluBench . As shown in Figure 2 , we organize data from SpatialKG into a systematic benchmark with 3 first-level categories and 5 second-level categories. GeoHaluBench consists of several branches, each corresponding to the regions of a city. Each city includes 21 tasks with a total of 2,100 instances. 2.2.2 Spatial World Hallucination Taxonomy The concept of hallucination traces its roots to the fields of pathology and psychology Macpherson and Platchias 2013 . Within the realm of NLP, hallucination is typically referred to as a