RenderMem: Rendering as Spatial Memory Retrieval

JooHyun Park, HyeongYeop Kang · 2026

Read RenderMem: Rendering as Spatial Memory Retrieval by JooHyun Park, HyeongYeop Kang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-14669.txt.

Introduction

Park and Kang treat rendering as memory retrieval for embodied agents in RenderMem. What an agent can see depends on where it stands. Stored photos and object lists ground spatial claims poorly. RenderMem keeps a 3D scene model instead of fixed views. Queries trigger fresh renders from the viewpoint the question implies. Line-of-sight, visibility and occlusion can then be judged from any stance. The frame fits standard vision-language models with no changes. AI2-THOR tests show steady gains on viewpoint queries, they report.

Catalog record

CREATORJooHyun Park, HyeongYeop Kang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36572
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.14669

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RenderMem: Rendering as Spatial Memory Retrieval JooHyun Park, HyeongYeop Kang Abstract Embodied reasoning is inherently viewpoint-dependent: what is visible, occluded, or reachable depends critically on where the agent stands. However, existing spatial memory systems for embodied agents typically store either multi-view observations or object-centric abstractions, making it difficult to perform reasoning with explicit geometric grounding. We introduce RenderMem , a spatial memory framework that treats rendering as the interface between 3D world representations and spatial reasoning. Instead of storing fixed observations, RenderMem maintains a 3D scene representation and generates query-conditioned visual evidence by rendering the scene from viewpoints implied by the query. This enables embodied agents to reason directly about line-of-sight, visibility, and occlusion from arbitrary perspectives. RenderMem is fully compatible with existing vision–language models and requires no modification to standard architectures. Experiments in the AI2-THOR environment show consistent improvements on viewpoint-dependent visibility and occlusion queries over prior memory baselines. Figure 1 : RenderMem retrieves spatial evidence from a stored 3D scene by rendering query-conditioned views, which serve as visual memory readouts for vision–language reasoning about visibility and object state. 1 Introduction Embodied reasoning is fundamentally viewpoint-dependent. In the physical world, perception is spatially situated: what an agent can observe depends on where it stands, and the same scene can yield entirely different evidence under different viewpoints. Because action decisions are grounded in what is visible from a specific location, visibility directly constrains movement, manipulation, and task execution. Viewpoint-conditioned reasoning is therefore intrinsic to embodied intelligence. Consider a robot responsible for inspecting safety conditions in a building. To ensure emergency equipment can be located quickly, it may need to reason: “From the main corridor, is the fire extinguisher visible, or is it occluded by the cabinet?” Answering such questions requires interpreting the corridor as a reference viewpoint and evaluating line-of-sight between objects. This is not merely object recognition—it is geometric reasoning over spatial configuration. Despite recent progress, existing spatial memory architectures struggle with such viewpoint-conditioned queries. Current approaches largely fall into three paradigms, each sharing a structural limitation. View-based memory [ 43 , 39 ] stores observations from discrete viewpoints and retrieves images for downstream reasoning. While effective for recognizing visible attributes, it is constrained by previously captured views and cannot reliably answer queries requiring novel or object-centric viewpoints. Object-centric memory [ 3 , 10 ] represents scenes as collections of objects and relations. Although compact and suitable for relational reasoning, it typically lacks explicit modeling of camera pose and line-of-sight geometry, making visibility reasoning difficult. 3D scene representations [ 44 , 13 ] encode richer geometric structure through meshes, volumetric maps, or neural fields. Yet their high-dimensional nature makes direct integration with language models challenging, often weakening the link between geometry and reasoning. These limitations point to a deeper issue: existing memory systems retrieve stored observations or abstract relations, but do not actively generate viewpoint-specific visual evidence required by a query. We argue that resolving viewpoint-dependent reasoning requires rethinking memory access itself. In this work, we introduce RenderMem , a rendering-based spatial memory framework built on a simple but powerful principle: rendering is the read operation of 3D memory . RenderMem maintains a persistent 3D scene representation and, when faced with a viewpoint-conditioned query, performs query-conditioned rendering to synthesize exactly the visual evidence needed for reasoning, as illustrated in Fig. 1 . Instead of forcing language models to interpret raw 3D geometry, RenderMem translates geometry into images aligned with the query’s specified viewpoint, which are then processed by standard vision–language models. By turning rendering into a first-class memory primitive, RenderMem enables explicit, geometrically grounded visibility and occlusion reasoning that prior memory paradigms cannot reliably support. Importantly, this design is not limited to static scenes. Because memory is maintained as an updatable 3D representation, any modification to object state or geometry is immediately reflected in subsequent query-conditioned renderings. Thus, RenderMem naturally extends to dynamic environments, serving as a spatial memory that evolves consistently with scene changes rather than requiring explicit rewriting of stored observations. Our contributions are as follows: • We identify viewpoint-dependent visibility and occlusion reasoning as a fundamental and under-addressed bottleneck in embodied spatial memory. • We propose RenderMem, introducing the abstraction of rendering as a query-conditioned memory read operation, enabling geometrically grounded reasoning without modifying existing vision–language architectures. • We develop query-conditioned viewpoint synthesis strategies that explicitly support visibility and occlusion reasoning, bridging the gap between 3D geometry and language-based inference. • Experiments show that RenderMem achieves superior performance over view-based and object-centric memory baselines while remaining robust under simulated reconstruction artifacts (blur, ghosting, and bounding-box noise). 2 Related Work 2.1 Embodied Agents with Vision-Language Models Embodied agents aim to perceive, reason, and act in interactive 3D environments. Early embodied systems primarily relied on modular pipelines with task-specific perception, mapping, and planning components, particularly for navigation and object search tasks [ 2 , 11 , 5 ] . While effective in constrained settings, these approaches struggled to generalize to open-ended tasks due to rigid symbolic representations and limited semantic understanding. Recent advances in large language models (LLMs) and vision–language models (VLMs) have substantially reshaped embodied intelligence. Pretrained multimodal models such as CLIP [ 34 ] , BLIP [ 27 ] , BLIP-2 [ 26 ] , Flamingo [ 1 ] , and LLaVA [ 28 ] demonstrate strong zero-shot generalization across diverse visual and linguistic tasks. These models enable embodied agents to interpret high-level instructions, reason over observations, and generalize across environments without task-specific retraining. Building on these capabilities, recent works on embodied agents leverage language models for action planning and decision-making grounded in visual observations. LEO [ 16 ] introduces a multimodal embodied agent that integrates 3D perception with language models to perform diverse tasks such as grounding, reasoning, and navigation in 3D environments. STEVE [ 45 ] combines visual perception modules with LLM-based reasoning to generate action plans for complex tasks. Octopus [ 42 ] formulates embodied control as program generation, where a vision–language model produces executable action code and improves behavior through environmental feedback. Progress in this direction is driven by embodied navigation and question answering benchmarks. Benchmarks such as Embodied Question Answering (EQA) [ 7 ] , ScanQA [ 4 ] , SQA3D [ 29 ] , OpenEQA [ 30 ] , and EXPRESS-Bench [ 19 ] evaluate an agent’s ability to integrate perception, memory, and reasoning over long interaction horizons. Exploration-centric benchmarks such as Explore-EQA [ 35 ] and GOAT-Bench [ 22 ] further emphasize long-horizon cognition and memory utilization. However, enabling such long-horizon, grounded reasoning requires a spatial memory system that can reliably store, organize, and retrieve perceptual information. 2.2 Spatial Memory Systems for Embodied Agents Spatial memory is a central component of embodied reasoning. One line of work adopts view-based memory, where agents retain RGB or RGB-D observations captured during exploration. 3D-Mem [ 43 ] organizes multi-view observations into memory snapshots that capture co-visible objects and scene context for VLM-based reasoning. LMEE [ 39 ] stores observed images and object metadata that support exploration and question answering. However, reliance on previously observed viewpoints can fail when queries require reasoning from novel or object-centric perspectives. Object-centric memory representations provide more structured abstractions. Scene graph–based methods [ 3 , 36 ] represent

Read the complete text