Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited Note to reviewers: this paper is a blend of a “summary” paper and a “full paper” – it is inspired by by a paper published at COSIT-24, but we have conducted many further experiments since then as the numbe

Anthony G Cohn, Robert E Blackwell · 2025

Read Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited Note to reviewers: this paper is a blend of a “summary” paper and a “full paper” – it is inspired by by a paper published at COSIT-24, but we have conducted many further experiments since then as the numbe by Anthony G Cohn, Robert E Blackwell (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12059.txt.

Introduction

Anthony Cohn and Robert Blackwell test whether chatbots know their compass points, in a 2025 paper. The team built a test bank from set templates, each placing an agent in a scene where a cardinal direction must be worked out. Templates vary the way the agent moves and the voice of the telling, from first person to third. Twenty-eight large language models took the test.

Even the newest models, built to reason step by step, missed some items. No model got every scene right. Direction sense sounds basic, yet the results show it rests on shaky ground in these systems. The paper sums up and extends a talk from a 2024 space-reasoning meeting, adding many fresh runs since then. The authors cast the test as a sharp probe for builders who want machines that can handle space talk in daily use.

Catalog record

CREATORAnthony G Cohn, Robert E Blackwell
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-12059

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited Note to reviewers: this paper is a blend of a “summary” paper and a “full paper” – it is inspired by by a paper published at COSIT-24, but we have conducted many further experiments since then as the number and variety of LLMs has continued to multiply. Anthony G Cohn, Robert E Blackwell Abstract We investigate the abilities of 28 Large language Models (LLMs) to reason about cardinal directions (CDs) using a benchmark generated from a set of templates, extensively testing an LLM’s ability to determine the correct CD given a particular scenario. The templates allow for a number of degrees of variation such as means of locomotion of the agent involved, and whether set in the first, second or third person. Even the newer Large Reasoning Models are unable to reliably determine the correct CD for all questions. This paper summarises and extends earlier work presented at COSIT-24. 1 Introduction This paper summarises and extends an earlier paper which investigated the ability of Large Language Models (LLMs) to reason about cardinal directions ( Cohn and Blackwell 2024b ) . That paper introduced two benchmarks: a small dataset of 100 questions designed to test simple factual understanding of cardinal directions, and a more complex one, with 5760 questions, generated from a set of six templates which requires some spatial reasoning to answer correctly. Already in the earlier paper several LLMs were obtaining very good performance on the small dataset, and newer models do even better. Therefore this update concentrates solely on the larger dataset and documents the performance of newer LLMs on it, conducting a more thorough analysis overall. Many claims (e.g. ( Creswell and Shanahan 2022 ; Huang and Chang 2023 ; Kojima et al. 2022 ) ) have been made since the emergence of Large Language Models (LLMs) as to their ability to reason. Spatial reasoning is of particular interest since not only does it underlie a human’s ability to operate in the physical world, but also because LLMs are not embodied; so the question arises, have they nonetheless acquired an ability to reason about situations which might occur in the real world? This is the question we address here. Spatial reasoning in general encompasses many aspects including topology, distance, and direction. Here, we restrict ourselves to reasoning about cardinal directions (CDs). CDs are important for many reasons, e.g.: (i) successful navigation and route finding/following usually requires a fundamental understanding and ability to reason about CDs: CDs are crucial to success when using a map. Equally, giving directions often relies, at least in part, on the use of CDs. (ii) Understanding the geography of an area depends on understanding the relative CD of one location to another – migration patterns, climate differences and economic variations are often underpinned by CDs. (iii) Weather patterns are often heavily influenced by the direction of the prevailing wind. (iv) CDs often play a critical role in cultural and historical contexts, e.g. the alignment of the pyramids to the four CDs, or in certain languages – e.g. the aboriginal language Guugu Yimithirr has no words for left or right, and spatial information is mainly conveyed using CDs ( Haviland 1998 ) . In this paper we therefore analyse how well LLMs can reason about cardinal and intercardinal directions. We do this by automatically constructing a large set of questions based on templates, for which the correct answer has been pre-determined, and testing each LLM’s ability to answer the questions correctly. In the original paper, Large Reasoning Models (LRMs), in which the models have been trained to perform inference-time reasoning, had not yet emerged, though chain-of-thought prompting (CoT) ( Wei et al. 2022 ) was already popular 1 1 1 We did not explore CoT as we wanted to measure raw LLM performance, independent of any particular prompting strategy; we do not explore CoT prompting in this paper either. . In the intervening months, LRMs have generally become the most performant LLMs, and this paper provides an update on how LRMs perform on our dataset, as well as some other new LLMs, such as GPT-4.5 preview. Since many LLMs now claim to have reasoning capability, we use the strict definition that an LRM is one which reports the use of reasoning tokens in its response to prompts in our benchmark. In this paper we investigate the following LRMs: azure-o1-2024-12-17, azure-o3-mini-2025-01-31, azure-o3-mini-2025-01-31-high, azure-o4-mini-2025-04-16, and deepseek-reasoner. For futher information on LRMs see Xu et al. 2025 who provide a survey covering the emergence and architectures of LRMs. 2 Related Work Despite the rapidly growing amount of research into LLMs and their capabilities there has been relatively little devoted specifically to spatial and/or geographic reasoning, and none which has tested their ability to reason about CDs in the way we do here. Of the existing work, we note benchmarks such as StepGame ( Li et al. 2024a ; Shi et al. 2022 ) which aims to test an LLM’s ability to correctly determine the spatial relationship between two objects, given the spatial relations between a larger set of objects, and between 1 and 10 reasoning steps are required to correctly determine the result; the direction relationships are not exclusively CDs, but also include, for example “clock face directions” (B is in the three o’clock direction from C). Not surprisingly, performance deteriorates as the required number of steps increases. However, performance increases markedly when an LLM is used to translate from the English specification to a logical representation and symbolic reasoning is used to compute the relationship. It should also be noted that the setting in StepGame is rather artificial compared to the naturalistic settings in our dataset described below. The SpartQA dataset ( Mirzaee et al. 2021 ) is also focused on assessing spatial reasoning, but does not contain any CDs. The bAbI dataset ( Weston et al. 2016 ) has one task which tests CDs understanding, task 19, which contains 1000 training and 1000 test questions: each instance contains 5 facts stating CDs between two objects, and then a question asking about the relation between two of them. Other work ( Yamada et al. 2024 ) has investigated whether LLMs can acquire an understanding of a spatial environment from a turn-by-turn description of a route, with landmarks named at each turn; whilst the LLMs did perform reasonably well, the experiment did not involve any CDs, only left/right and up/down. There are different kinds of spatial reasoning tasks which can be considered. Relational composition is one of the most studied from a theoretical point of view. A composition table records the results for all combinations of relations in a particular spatial representation such as RCC; investigations ( Cohn 2023 ; Cohn and Blackwell 2024a ) into various LLMs’ abilities to compute all RCC compositions found reasonable accuracy levels (reduced when relations are anonymised); however RCC is a purely mereotopological calculus with no notion of direction embedded in its semantics. Some LLMs have been built specifically for geo-applications, but these do not focus on reasoning about directions but rather aspects such as toponym recognition, e.g. Li et al. 2023 . 3 Experimental Design Whilst testing

Read the complete text