LocationReasoner: Evaluating LLMs on Real-World Site Selection Reasoning

Miho KodaYu ZhengRuixian MaMingyang SunDevesh Pansare Fabio DuartePaolo Santi Masssachusetts Institute of Technology, Cambridge, MA USA {mihok485,yu_zheng,ruixian,msun14,dp777,fduarte,psanti}@mit.edu · 2025

Read LocationReasoner: Evaluating LLMs on Real-World Site Selection Reasoning by Miho KodaYu ZhengRuixian MaMingyang SunDevesh Pansare Fabio DuartePaolo Santi Masssachusetts Institute of Technology, Cambridge, MA USA {mihok485,yu_zheng,ruixian,msun14,dp777,fduarte,psanti}@mit.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-13841.txt.

Introduction

Miho Koda and co-authors build a benchmark in a 2025 paper to test language models on choosing real world sites. Picking a site for a shop or facility means juggling space, environment, and transport limits at once. Most reasoning tests use math or code, so they miss this kind of messy, real choice.

Their benchmark holds crafted queries at varied difficulty, backed by a sandbox with tools for checking limits. Tests on data from Boston, New York, and Tampa show top reasoning models gain little over older ones. Even the newest model tested failed on thirty percent of tasks. Step by step agent styles often did worse by overthinking. The authors release the data and code for public use.

Catalog record

CREATORMiho KodaYu ZhengRuixian MaMingyang SunDevesh Pansare Fabio DuartePaolo Santi Masssachusetts Institute of Technology, Cambridge, MA USA {mihok485,yu_zheng,ruixian,msun14,dp777,fduarte,psanti}@mit.edu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-13841

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LocationReasoner: Evaluating LLMs on Real-World Site Selection Reasoning Miho KodaYu ZhengRuixian MaMingyang SunDevesh Pansare Fabio DuartePaolo Santi Masssachusetts Institute of Technology, Cambridge, MA USA {mihok485,yu_zheng,ruixian,msun14,dp777,fduarte,psanti}@mit.edu Abstract Recent advances in large language models (LLMs), particularly those enhanced through reinforced post-training, have demonstrated impressive reasoning capabilities, as exemplified by models such as OpenAI o1 and DeepSeek-R1. However, these capabilities are predominantly benchmarked on domains like mathematical problem solving and code generation—leaving open the question of whether such reasoning skills generalize to complex, real-world scenarios. In this paper, we introduce LocationReasoner , a benchmark designed to evaluate LLMs’ reasoning abilities in the context of real-world site selection , where models must identify feasible locations by reasoning over diverse and complicated spatial, environmental, and logistical constraints. The benchmark comprises over 300 carefully crafted queries of varying difficulty levels, supported by a sandbox environment with in-house tools for constraint-based location search. Extensive evaluations reveal that state-of-the-art reasoning models offer limited improvement over their non-reasoning predecessors in real-world contexts, with even the latest OpenAI o4 model failing on 30% of site selection tasks. Moreover, agentic strategies such as ReAct and Reflexion often suffer from over-reasoning, leading to worse outcomes than direct code-generation prompting. With key limitations of LLMs in holistic and non-linear reasoning highlighted, we release LocationReasoner to foster the development of LLMs and agents capable of robust, grounded reasoning in real-world decision-making tasks. Codes and data for our benchmark are available at https://github.com/miho-koda/LocationReasoner . 1 Introduction Large language models (LLMs) have achieved tremendous success across a broad spectrum of tasks, ranging from human-like dialogue achiam2023gpt ; team2023gemini ; liu2024deepseek ; bai2023qwen ; glm2024chatglm ; touvron2023llama to agent-based simulations park2023generative ; shanahan2023role ; wang2024survey –progress largely driven by scaling data and compute sutton2019bitter ; wei2022emergent . Recent advances extend the notion of scaling beyond training, introducing inference-time scaling techniques such as chain-of-thought (CoT) prompting wei2022chain and agentic strategies like ReAct yao2023react , which enable deeper reasoning through intermediate steps and tool use. In particular, large-scale reinforcement learning has been applied during post-training to generate high-quality reasoning traces (e.g., CoT), giving rise to a new class of reasoning models such as OpenAI o1 jaech2024openai and DeepSeek-R1 guo2025deepseek . These reasoning capabilities are critical for enabling LLMs to tackle complex, real-world tasks that often require decomposing problems, exploring alternative strategies, and interacting with external tools. Benchmarks play a crucial role in driving the progress of LLMs patterson2012better , and the rapid advancement of these models has created a growing demand for more challenging and informative evaluation tasks. While general language understanding benchmarks are widespread chiang2024chatbot ; wang2024mmlu ; hendrycks2021mmlu ; rein2024gpqa ; huang2023c , assessing reasoning capabilities remains considerably more difficult—primarily due to the challenge of verifying the correctness of LLMs’ reasoning outputs. As a result, current reasoning models are predominantly evaluated on domains with easily verifiable solutions, where verifying a solution is much simpler than generating one. This includes domains such as mathematical problem solving and programming, exemplified by benchmarks like AIME aime2024 , MATH500 math500 , and CodeForces quan2025codeelo . In contrast, few benchmarks have tackled real-world reasoning scenarios, which often involve more complicated constraints than those found in math or coding. For instance, the TravelPlanner benchmark xie2024travelplanner evaluates LLM agents on practical planning tasks but does not include recent reasoning models and lacks automated verification mechanisms, relying heavily on manual annotation. Consequently, it remains unclear whether current reasoning LLMs and agents can effectively generalize to real-world reasoning tasks. This gap highlights the urgent need for a new benchmark that targets real-world reasoning while remaining scalable and annotation-free. In this work, we introduce LocationReasoner , a benchmark designed to evaluate the real-world reasoning capabilities of LLMs through the task of site selection . Site selection is a common decision-making process in practical business settings, where the objective is to identify feasible locations based on a diverse set of criteria. This task presents a significant reasoning challenge, as it requires LLMs to perform structured location search and filtering over queries that involve multiple constraints with complex logical dependencies. For instance, selecting a site for a new restaurant may demand reasoning over temporal consumption constraints and spatial transportation constraints, with various conditions often combined through Boolean logic. To comprehensively assess LLMs’ ability to reason under such practical and multifaceted conditions, we construct the LocationReasoner benchmark, which includes over 300 site selection queries spanning a wide range of difficulty levels. The benchmark is supported by a sandbox environment equipped with offline datasets and in-house tools for constraint-based location search. LocationReasoner differs fundamentally from existing benchmarks in two key aspects. First, it focuses on practical reasoning, where LLMs must decompose complex constraints into smaller executable steps and generate codes using available tools to solve each query. Second, the benchmark features deterministic and automatically verifiable queries, enabling fully scalable evaluation without human annotation, and allowing seamless extension to much larger benchmark suites. We summarize the key contributions of this work as follows: • We introduce LocationReasoner, a real-world reasoning benchmark that grounds LLMs in practical site selection tasks. The benchmark includes a suite of curated datasets, toolsets, and a sandbox environment to enable out-of-the-box evaluation of LLMs in complex constraint-based reasoning. • We conduct a comprehensive evaluation of four major LLM families–OpenAI jaech2024openai , Gemini team2023gemini , Claude anthropic2024claude35sonnet , and DeepSeek guo2025deepseek –covering both general-purpose and reasoning-augmented models, and two representative agentic workflows, ReAct yao2023react and Reflexion shinn2023reflexion . • Our benchmarking results reveal that current LLMs struggle to effectively handle real-world reasoning challenges. For instance, the latest OpenAI o4 model achieves only 69.99% success on site selection queries. Furthermore, reasoning-augmented models and agentic strategies provide limited gains compared to direct prompting of general models for code generation. In-depth analysis uncovers key limitations in holistic and non-linear reasoning, pointing to critical bottlenecks that need to be addressed to improve LLM performance on practical, real-world tasks. 2 LocationReasoner Figure 1 : Query generation and execution pipeline. Queries are automatically generated using predefined constraints, ranges, thresholds, and logic composition—either through rule-based or LLM-based methods. Each query is executed via a deterministic code path using interdependent in-house tools or through an LLM agent. The results are then compared to evaluate whether the agent produces the correct filtered zones. We center the LocationReasoner benchmark around practical site selection, a problem that is easy to understand and intuitive, but also requires multi-step reasoning across spacial, economic, and infrastructural constraints. Built on real data from Boston, MA and a fixed set of in-house tools, the benchmark offers a controlled and interpretable environment to test LLMs’ ability to reason and plan under real-world constraints. As illustrated in Figure 1 , we construct the entire query generation and execution pipeline such that it can be fully automated to support scalable evaluation. By programmatically defining constraints, threshold ranges, and logical compositions, queries can be generated in bulk through either rule-based generation or LLM-based generation. The rule-based approach is rigid and syntactically precise as predefined constraints are directly encoded into executable templates. It ensures that all conditions are logically well-formed and interpretable by deterministic code. LLM-based generation introduces linguistic diversity. The system feeds the same set of structured constraints to a language model and asks it to produce free-form queries

Read the complete text