Rethinking the Simulation vs. Rendering Dichotomy: No Free Lunch in Spatial World Modelling

Dezhi Luo, Qingying Gao, Hokin Deng · 2025

Read Rethinking the Simulation vs. Rendering Dichotomy: No Free Lunch in Spatial World Modelling by Dezhi Luo, Qingying Gao, Hokin Deng (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-20835.txt.

Introduction

Dezhi Luo, Qingying Gao, and Hokin Deng revisit a split in how minds and machines are said to handle space, by inner simulation or by rendered scenes. Borrowed accounts from human cognition are often vague, and the authors use evidence from aphantasia, where people lack voluntary mental images, to sharpen the debate.

Work on the brain basis of visual awareness suggests that simulation and felt perception share structured maps of relations between things. Rich perceptual detail also helps embodied AI systems act better in physical settings, backing the shared map view. The team calls for machine designs that keep structured perceptual records as a core part of spatial thought, rather than treating images as mere decoration.

Catalog record

CREATORDezhi Luo, Qingying Gao, Hokin Deng
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-20835

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rethinking the Simulation vs. Rendering Dichotomy: No Free Lunch in Spatial World Modelling Dezhi Luo, Qingying Gao, Hokin Deng Abstract Spatial world models, representations that support flexible reasoning about spatial relations, are central to developing computational models that could operate in the physical world, but their precise mechanistic underpinnings are nuanced by the borrowing of underspecified or misguided accounts of human cognition. This paper revisits the simulation versus rendering dichotomy and draws on evidence from aphantasia to argue that fine-grained perceptual content is critical for model-based spatial reasoning. Drawing on recent research into the neural basis of visual awareness, we propose that spatial simulation and perceptual experience depend on shared representational geometries captured by higher-order indices of perceptual relations. We argue that recent developments in embodied AI support this claim, where rich perceptual details improve performance on physics-based world engagements. To this end, we call for the development of architectures capable of maintaining structured perceptual representations as a step toward spatial world modelling in AI. 1 Introduction Human reasoning is distinguished by its capacity to construct structured simulations of the external world, or world models, that support prediction, planning, and counterfactual inference beyond what is immediately perceived ( Johnson-Laird 1983 ; Wooldridge and Jennings 1995 ; LeCun 2022 ) . Among these, spatial world models serve as a foundational substrate for structured spatial reasoning: the ability to represent and manipulate spatial relations in a flexible, generalizable manner ( Kuipers 1978 ; Kosslyn et al. 1979 ; Hegarty 2011 ; Gao et al. 2025 ) . Recently, spatial world modelling has garnered significant attention as a promising approach for enhancing spatial reasoning capabilities in foundation models ( Chaplot et al. 2021 ; Spies et al. 2024 ; Gao et al. 2025 ; Cai et al. 2025 ) . Yet, the underlying computational mechanisms remain poorly understood, in part because the concept of a spatial world model is often treated as an idealized abstraction rather than an empirically grounded construct. Current evaluation methods for model-based spatial reasoning tend to rely on task paradigms that are assumed to require internal simulation, such as mental rotation, visual perspective-taking, and mechanical reasoning ( Jelassi et al. 2022 ; Gao et al. 2024 ; Sun et al. 2024 ; Zhang et al. 2024 ; Wang et al. 2025a ) . These assumptions are often informed by cognitive science frameworks that describe how humans approach spatial problems ( Shepard and Metzler 1971 ; Hegarty 2004 ) . If a task is believed to necessitate mental simulation of spatial transformations, it is categorized as diagnostic of model-based reasoning. However, such paradigms tend to ignore the nuanced nature of the internal mechanisms humans employ to navigate these problems, which await consensus among cognitive scientists. This paper addresses this important challenge of formulating a clear conceptualization of spatial world models: namely, what kinds of internal representations are required to sustain robust simulations of the external world’s spatial properties? To this end, we first revisit the so-called “simulation vs. rendering” dichotomy in mental imagery research. Examining the case of aphantasia, we argue that the standard formulation of this dichotomy is misconstrued. Rather than treating simulation and rendering as separate processes ( Balaban and Ullman 2025 ) , we propose that both may rely on convergent, higher-order representations that capture the instantiated relations between perceptual objects. The implication is that the representational substrate of spatial world models must be more fine-grained than previously assumed, requiring architectures that go beyond shallow modelling of spatial relations. 2 On the Split Between Simulation and Rendering Efforts to characterize spatial world models often appeal to how humans appear to solve tasks that presumably require internal simulation. A canonical example is mental rotation, where response times scale with angular disparity, suggesting incremental manipulation of internal spatial representations ( Pylyshyn 1979 ; Khooshabeh et al. 2013 ; Hilton et al. 2022 ) . This has been interpreted as evidence for simulation-based reasoning ( Pylyshyn 1979 ; Searle and Hamm 2017 ) . However, the underlying mechanisms are not directly observable, and mental rotation is typically accompanied by conscious visual experience , raising the question of whether such imagery plays a functional role or is merely epiphenomenal ( Wexler et al. 1998 ; Vingerhoets et al. 2002 ) . To address this, recent proposals distinguish between physics-based and graphics-based components of mental imagery ( Balaban and Ullman 2025 ) . On this view, tasks such as rotation, scanning, and manipulation may be solved through simulations over structured spatial representations, without requiring perceptual rendering . Rendering may still occur, but it bares no direct relevance to correctly and robustly completing these tasks. Here, we note that this position hinges on a linear interpretation of the functional specialization across streams of visual processing in the human brain, and that it is misguided. Below, we discuss the theoretical claims and corresponding evidence in objection to the above interpretation, and highlight that they instead point toward a non-linear, hierarchical relationship between simulation and rendering. 2.1 Examining the Spatial Imagery Framework and the Case of Aphantasia The proposed simulation vs. rendering dichotomy draws particular support from studies of aphantasia . Individuals with aphantasia report an inability to generate voluntary visual images, yet often perform comparably to the general population on tasks such as mental rotation ( Kay et al. 2024 ; Balaban and Ullman 2025 ) . This has been interpreted as evidence that spatial simulation can operate independently of conscious vision. To explain this, researchers have proposed a distinction between spatial imagery and visual or object-based imagery, with the former supporting abstract reasoning about relative positions, distances, and spatial relations without relying on pictorial content. On this account, preserved task performance in aphantasia is attributed to intact spatial imagery mechanisms. Balaban and Ullman 2025 further has broadened this concept to encompass physical simulation for reasoning about motion and interaction. Under this conceptualization, spatial imagery encodes the entire scene—including geometric structure, spatial extent, and other object properties—sufficient to simulate its temporal evolution. According to them, while no rendering occurs, non-physical attributes can still be abstractly represented, akin to an engineer’s amodal model. Importantly, this implies that simulations required for capturing the spatial relations of the real-world environment, sufficient for structured physical reasoning, do not demand the computations used to support the fine-grained perceptual content in the conscious visual scene that someone without aphantasia would experience when completing tasks. While the spatial imagery framework offers a plausible account of preserved task performance in aphantasia ( Kay et al. 2024 ) , it rests on a notion that remains under-specified. Notably, it is not apparent what exactly constitutes the content of spatial "imagery". Proponents of the framework often suggest that spatial imagery involves abstracted or schematic representations of spatial features, such as location, relation, or shape, that are "modality-neutral" and potentially amodal. Phillips 2025 , for example, describes such imagery as “neutral as to whether the location, relation, shape or structure is imagined as seen or touched.” Nevertheless, when describing how such abstract "imageries" are deployed in action, an example is given as the "subjects imagine grasping the shape and rotating it". This framing seems to rely on the covert recruitment of embodied sensory modalities without

Read the complete text