From Pixels to BFS: High Maze Accuracy Does Not Imply Visual Planning

Alberto G. Rodriguez Salgado · 2026

Read From Pixels to BFS: High Maze Accuracy Does Not Imply Visual Planning by Alberto G. Rodriguez Salgado (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26839.txt.

Introduction

Rodriguez introduces MazeBench, a set of 110 generated maze images for testing multimodal models. Top systems score high: GPT-5.4 reaches 91 percent and Gemini 3.1 Pro 79 percent. The traces tell a different story, though. Models turn images into text grids, then search step by step in tokens. That brute-force style uses thousands of tokens per maze. Given a correct text grid, one model jumped from 6 to 80 percent. Weak visual extraction, not search, is the core gap.

Catalog record

CREATORAlberto G. Rodriguez Salgado
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34577
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26839

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Pixels to BFS: High Maze Accuracy Does Not Imply Visual Planning Alberto G. Rodriguez Salgado Abstract How do multimodal models solve visual spatial tasks—through genuine planning, or through brute-force search in token space? We introduce MazeBench , a benchmark of 110 procedurally generated maze images across nine controlled groups, and evaluate 16 model configurations from OpenAI, Anthropic, Google, and Alibaba. GPT-5.4 solves 91% and Gemini 3.1 Pro 79%, but these scores are misleading: models typically translate images into text grids and then enumerate paths step by step, consuming 1,710–22,818 tokens per solve for a task humans do quickly. Without added reasoning budgets, all configurations score only 2–12%; on 20 × \times 20 ultra-hard mazes, they hit token limits and fail. Qualitative traces reveal a common two-stage strategy: image-to-grid translation followed by token-level search, effectively BFS in prose. A text-grid ablation shows Claude Sonnet 4.6 rising from 6% on images to 80% when given the correct grid, isolating weak visual extraction from downstream search. When explicitly instructed not to construct a grid or perform graph search, models still revert to the same enumeration strategy. MazeBench therefore shows that high accuracy on visual planning tasks does not imply human-like spatial understanding. Code and mazes are available at https://github.com/alrod97/LLMs_mazes . 1 Introduction Multimodal large language models (MLLMs) have achieved impressive performance across a wide range of vision-language tasks, from visual question answering to diagram understanding and mathematical reasoning in visual contexts Lu et al. 2024 ; Yue et al. 2024 ; Fu et al. 2024a . Recent benchmarks have begun probing deeper visual capabilities, finding that models struggle with tasks that humans—even young children—solve effortlessly Fu et al. 2024b ; Tong et al. 2024 ; Chen et al. 2026 . Yet when models do score well on such tasks, a crucial question remains: does a high accuracy score mean the model actually understands the task, or is it achieving the right answer through a fundamentally different—and far less efficient—mechanism? We investigate this question through visual maze solving, a task that is conceptually simple for humans: given a pixel-art maze image, find the shortest path from the player to the treasure. A human glances at even a complex 20 \times 20 maze and traces the path visually in seconds. We find that frontier MLLMs can also solve many of these mazes—GPT-5.4 achieves 91%, Gemini 3.1 Pro 79%—but they do so in a qualitatively different way. Rather than spatial planning, models translate the image into a token-level grid representation and then perform serial path enumeration : they brute-force the solution step by step in natural language, consuming thousands of reasoning tokens for a task that requires no deliberation for a human. When the path is too long to enumerate within the token budget, the model gives up—not because it cannot see the maze, but because it runs out of space to think. This finding has implications beyond mazes. It suggests that benchmark accuracy alone can be misleading about the nature of model capabilities: a model may score 90% on a task while using a fundamentally different—and far more costly—cognitive strategy than the one the benchmark was designed to measure. We design MazeBench with three properties that make this analysis possible: Controlled difficulty via procedural generation. We build a procedural maze generator producing mazes with controlled grid size (5 \times 5 to 20 \times 20), wall density (0–55%), trap count (0–25), border walls, and varied start/goal positions. Ground-truth shortest paths are computed via BFS. The 110 mazes are organized into nine experimental groups—including diagnostic, grid scale, wall density, trap ablation, and ultra-hard—enabling clean ablation studies (Figure 1). Reasoning effort as a controlled variable. We systematically vary the reasoning budget via frontier API controls (OpenAI’s reasoning_effort , Anthropic’s adaptive thinking), producing a scaling curve from no thinking to medium effort on the same visual inputs. Token efficiency as a window into strategy. We report the total tokens consumed (thinking + output) per solve. This reveals how models solve mazes, not just whether they do: GPT-5.4 low requires 1,710 tokens per solve, while Gemini 3 Flash uses 15,171—both achieving correct answers but through vastly different amounts of brute-force enumeration (Figure 1). Our main findings are: 1. High scores mask brute-force strategies. GPT-5.4 solves 91% of mazes, but consumes 2,913 tokens per solve—serial enumeration, with no observable spatial planning behavior. On 20 \times 20 mazes where paths exceed the token budget, accuracy drops to 30% (Section 5.6 ). 2. Without added reasoning budgets, performance remains very low. All 16 configurations score only 2–12% in their no-thinking or lowest-budget settings, even when some parse the grid correctly. For the stronger models, the failure is primarily in path planning rather than basic visual parsing (Section 5.3 ). 3. The same task, radically different costs. Models vary by 9 \times in tokens per solve (1,710 for GPT-5.4 vs. 15,171 for Gemini 3 Flash), revealing that “solving a maze” means very different things computationally across providers (Section 5.5 ). 4. Model size does not buy spatial reasoning. Claude Opus 4.6 (the largest Anthropic model) solves the same 4% as Haiku 4.5 (the smallest). Spatial planning does not appear to be an emergent property of scale (Section 6 ). 2 Related Work Visual perception gaps in MLLMs. Several recent benchmarks have documented systematic failures in visual perception. BLINK Fu et al. 2024b reformats 14 classic computer vision tasks as multiple-choice questions and finds that GPT-4V achieves only 51% versus 96% for humans, concluding that perception tasks “resist mediation through natural language.” Tong et al. 2024 identify “CLIP-blind pairs”—images that vision encoders conflate despite clear visual differences—and construct the MMVP benchmark exposing failures on basic visual patterns. BabyVision Chen et al. 2026 tests core visual abilities that human children master by age 3–6, finding that even Gemini 3 Pro scores only 49.7 versus 94.1 for adults. Notably, BabyVision includes a maze-tracing task in which models select which entrance connects to an exit from a multiple-choice list—a perceptual tracking task. Our benchmark differs fundamentally: rather than choosing among predefined options, models must generate the complete shortest path as an exact sequence of moves (U/D/L/R), requiring both visual parsing and multi-step spatial planning. This distinction allows us to show that failures compound across both stages: some models (Claude) fail primarily at visual grid extraction, while others (GPT-5.4, Gemini) parse the grid correctly but still resort to brute-force token enumeration rather than spatial planning. Multimodal reasoning benchmarks. MathVista Lu et al. 2024 and MathVerse Zhang et al. 2024 evaluate mathematical reasoning in visual contexts, finding that models often rely on textual cues rather than diagram understanding. EMMA Hao et al. 2025 tests cross-modal reasoning across math, physics, chemistry, and coding, reporting that even chain-of-thought prompting and test-time compute scaling underperform (ICML 2025). MMMU Yue et al. 2024 and MME Fu et al. 2024a provide comprehensive evaluation suites spanning dozens of disciplines, while MMEvalPro Huang et al. 2025 addresses systematic biases in multiple-choice evaluation by introducing perception prerequisite questions. Chen et al. 2024b audit evaluation methodology itself, questioning whether current benchmarks measure the capabilities they claim to. Our benchmark complements this body of work by targeting a single, tightly controlled task—visual pathfinding—that isolates spatial reasoning from domain knowledge.

Read the complete text