UrbanPlanBench: A Comprehensive Urban Planning Benchmark for Evaluating Large Language Models

Yu Zheng, Longyi Liu, Yuming Lin, Jie Feng, Guozhen Zhang, Depeng Jin, Yong Li · 2025

Read UrbanPlanBench: A Comprehensive Urban Planning Benchmark for Evaluating Large Language Models by Yu Zheng, Longyi Liu, Yuming Lin, Jie Feng, Guozhen Zhang, Depeng Jin, Yong Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21027.txt.

Introduction

Yu Zheng and co-authors built a test of how well large language models know city planning, in this 2025 paper. Planning shapes streets, homes, and public space, and it rests on deep expert knowledge. Their test covers basic ideas, trade knowledge, and rules and management, matched to what human planners must know.

Most models scored far below trade standards, with rules the weakest area for about 70 percent of them. The team also built a training set of more than 30,000 question pairs from exams and textbooks. Extra training raised scores on recall and grasp of planning facts. Gains were smaller on tasks that need trade terms and careful thought. All tools were shared in public for later work.

Catalog record

CREATORYu Zheng, Longyi Liu, Yuming Lin, Jie Feng, Guozhen Zhang, Depeng Jin, Yong Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-21027

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

UrbanPlanBench: A Comprehensive Urban Planning Benchmark for Evaluating Large Language Models Yu Zheng, Longyi Liu, Yuming Lin, Jie Feng, Guozhen Zhang, Depeng Jin, Yong Li Abstract . The advent of Large Language Models (LLMs) holds promise for revolutionizing various fields traditionally dominated by human expertise. Urban planning, a professional discipline that fundamentally shapes our daily surroundings, is one such field heavily relying on multifaceted domain knowledge and experience of human experts. The extent to which LLMs can assist human practitioners in urban planning remains largely unexplored. In this paper, we introduce a comprehensive benchmark, UrbanPlanBench, tailored to evaluate the efficacy of LLMs in urban planning, which encompasses fundamental principles, professional knowledge, and management and regulations, aligning closely with the qualifications expected of human planners. Through extensive evaluation, we reveal a significant imbalance in the acquisition of planning knowledge among LLMs, with even the most proficient models falling short of meeting professional standards. For instance, we observe that 70% of LLMs achieve subpar performance in understanding planning regulations compared to other aspects. Besides the benchmark, we present the largest-ever supervised fine-tuning (SFT) dataset, UrbanPlanText, comprising over 30,000 instruction pairs sourced from urban planning exams and textbooks. Our findings demonstrate that fine-tuned models exhibit enhanced performance in memorization tests and comprehension of urban planning knowledge, while there exists significant room for improvement, particularly in tasks requiring domain-specific terminology and reasoning. By making our benchmark, dataset, and associated evaluation and fine-tuning toolsets publicly available at https://github.com/tsinghua-fib-lab/PlanBench , we aim to catalyze the integration of LLMs into practical urban planning, fostering a symbiotic collaboration between human expertise and machine intelligence. DOI: XXXXXXX.XXXXXXX Conference: ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 03–07, 2025; Toronto, ON, Canada ISBN: 978-1-4503-XXXX-X/2018/06 1. Introduction Recent breakthroughs in Large Language Models (LLMs) ( Touvron et al., 2023 ; Zeng et al., 2023 ) have showcased remarkable capabilities in generating text, reasoning, and knowledge QA, unlocking a plethora of applications ranging from chatbots ( OpenAI, 2022 ) to programming copilots ( Chen et al., 2021 ) . Besides general-purpose evaluation, assessing their capabilities in specialized domains is crucial for understanding the real-world impact of LLMs ( Chen and Deng, 2023 ; Koncel-Kedziorski et al., 2023 ; Fei et al., 2023 ; Liu et al., 2024 ) . In this paper, we focus on one critical field, urban planning, which stands as a cornerstone in shaping modern city life, yielding profound influence on over 4 billion urban residents worldwide. Urban planning is a complex endeavor that intertwines various disciplines, demanding a deep understanding of domain knowledge. Despite the advent of technological advancements, the field continues to heavily rely on the expertise and experience of human planners. For instance, human planners devote substantial time to tasks such as planning text management, review, and assessment ( Zhu et al., 2024 ) . Moreover, the limitations inherent in human experience often lead to errors and inefficiencies in planning outcomes ( Zheng et al., 2023a ) . Notably, the integration of LLMs in urban planning contexts has emerged as a promising avenue, leveraging their pre-trained world knowledge to tackle complex computational tasks ( Zhou et al., 2024 ; Fu et al., 2024 ; Xu et al., 2023 ; Zhu et al., 2024 ; Li et al., 2024 ) . However, the inherent challenges of hallucination and vagueness present significant hurdles, particularly when addressing specialized problems within urban planning ( Zhang et al., 2023 ) . While various benchmarks such as SuperGLUE ( Wang et al., 2019 ) , BIG-BENCH ( Srivastava et al., 2022 ) and C-Eval ( Huang et al., 2023 ) have been proposed to evaluate LLM effectiveness in understanding and solving intricate tasks, the absence of dedicated benchmarks for urban planning restricts our ability to quantify the extent to which LLMs acquire specialized knowledge and their potential to enhance the productivity of human planners. As a human-centered field, matching the performance of human planners marks a milestone for LLMs and signifies their mastery of urban planning capabilities. In alignment with the rigorous standards set by the certified urban planner qualification examination in China, we introduce UrbanPlanBench, a comprehensive benchmark designed to evaluate LLMs across various perspectives of urban planning, including fundamental principles, professional knowledge, and management and regulations. The benchmark mirrors the latest available examination standards as of 2022, enabling a comparative analysis between LLM and human planners, shedding lights on whether current general-purpose LLMs attain a human-level understanding of urban planning. Leveraging this benchmark, we scrutinize recent open-source LLMs, including LLaMA1/2/3/3.1 ( Touvron et al., 2023 ; Meta, 2024 ) , Gemma1/2 ( Team et al., 2024a ; Team et al., 2024b ) , ChatGLM3/4 ( GLM et al., 2024 ) , Baichuan2 ( Baichuan, 2023 ) , Qwen1.5/2 ( Bai et al., 2023 ; Yang et al., 2024 ) , and Yi ( Young et al., 2024 ) , as well as commercial LLMs like ChatGPT 3.5/4o, to assess their acquisition of planning skills. Additionally, we also evaluate the effect of prompting techniques for LLMs including chain of thought (COT) ( Wei et al., 2022b ) and retrieval augmented generation (RAG) ( Lewis et al., 2020 ; Gao et al., 2023 ) . Supervised fine-tuning (SFT) stands as a prevalent method to build domain-specific LLMs. However, to our knowledge, there are currently no off-the-shelf resources for fine-tuning LLMs specifically for urban planning. This gap arises from the significant disparity between the distribution of urban planning knowledge in descriptive texts and the required form of SFT data, which necessitates sample pairs comprising instructions and responses. To bridge this gap, we further introduce UrbanPlanText, the largest-ever dataset tailored for SFT of LLMs in urban planning. Comprising over 30,000 instruction pairs derived from textbooks and past exams, UrbanPlanText serves as a comprehensive collection of specialized urban planning contents. We conduct extensive experiments to assess current advanced LLMs on UrbanPlanBench. While LLMs demonstrate significantly better performance than random guessing, there remains large room for improvement, indicating a limited mastery of urban planning skills. Notably, most of the current LLMs can not surpass the certification bar of the urban planner qualification examination, which roughly represents the top 10% proficiency level of human planners. Additionally, our analysis highlights an imbalance in LLM performance across three key urban planning perspectives: they exhibit greater proficiency in understanding planning principles and knowledge but tend to falter in memorizing regulations, leading to factual errors. Moreover, we find that finetuning LLMs with UrbanPlanText can effectively enhance their ability to answer urban planning-related questions. By introducing both UrbanPlanBench and UrbanPlanText, we aim to facilitate the seamless integration of LLMs into practical urban planning workflows, thereby lowering entry barriers for practitioners and enabling them to fully leverage advanced AI tools in their work. Figure 1. Example questions of UrbanPlanBench. MCQ-S has four options where only one option is correct. MCQ-M is more challenging, featuring two to four correct options from a total of five options. Contents are translated from Chinese to English. 2. Benchmarking LLMs on Urban Planning Table 1. Accuracy (%) of LLMs on three subjects of UrbanPlanBench. S and M indicate MCQs with one single correct answer and multiple correct answers, respectively. Full represents the overall accuracy of both

Read the complete text