Cube Bench: A Benchmark for Spatial Visual Reasoning in MLLMs

Dhruv Anand, Ehsan Shareghi · 2025

Read Cube Bench: A Benchmark for Spatial Visual Reasoning in MLLMs by Dhruv Anand, Ehsan Shareghi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20595.txt.

Introduction

Dhruv Anand and Ehsan Shareghi built a benchmark out of a twisty puzzle. Cube Bench tests multimodal language models on Rubik's cube tasks, splitting spatial and sequential reasoning into five skills. Models must reconstruct cube faces from images and text, pick the best next move, and predict a move's effect without making it. They must also execute multi-step plans while recovering from errors, and catch their own mistakes.

Seven recent models faced identical scrambled states, prompts, and scoring. Accuracy collapsed as scramble depth grew, and once a solution trajectory stalled, models almost never recovered. Good face reconstruction did not imply good move choice. The strongest closed model led throughout, open models hovered near chance on hard settings, and reflective self-correction helped only modestly.

Catalog record

CREATORDhruv Anand, Ehsan Shareghi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE81 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-20595

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Cube Bench: A Benchmark for Spatial Visual Reasoning in MLLMs Dhruv Anand, Ehsan Shareghi Abstract We introduce Cube Bench , a Rubik’s-cube benchmark for evaluating spatial and sequential reasoning in multimodal large language models (MLLMs). The benchmark decomposes performance into five skills: (i) reconstructing cube faces from images and text, (ii) choosing the optimal next move, (iii) predicting the outcome of a candidate move without applying it, (iv) executing multi-step plans while recovering from mistakes, and (v) detecting and revising one’s own errors. Using a shared set of scrambled cube states, identical prompts and parsers, and a single distance-to-solved metric, we compare recent MLLMs side by side as a function of scramble depth. Across seven MLLMs, accuracy drops sharply with depth; once a trajectory stalls or diverges, models rarely recover, and high face-reconstruction accuracy does not guarantee competent action selection or multi-step execution. A pronounced closed- vs open-source gap emerges: the strongest closed model leads on both single-step perception tasks and multi-step control tasks, while open-weight models cluster near chance on the hardest settings; yet even the best MLLM degrades at higher cube complexity. A simple self-correction via reflective thinking yields modest gains but can also introduce overthinking. Cube Bench offers a compact, reproducible probe of sequential spatial reasoning in MLLMs. 1 1 1 The code is available at https://github.com/dana-23/cube-bench . 1 Introduction Multimodal large language models (MLLMs) now match or surpass prior systems on static perception benchmarks, i.e., recognizing objects, transcribing text in images, and answering short questions. However, practical deployments require models that can act interactively: taking actions, observing consequences, and staying coherent across many steps (closed-loop control). In such settings, small errors compound and recovery is rare; single-shot vision tests provide little signal about whether a model can plan, act, and correct itself. Surveys of multimodal benchmarks highlight this mismatch between static leaderboards and the demands of interactive use 13 , and recent web-based agent evaluations similarly report that strong snapshot performance does not directly translate to reliable multi-step behaviour 31 ; 10 . Classical work on sequential decision making has long emphasized how compounding errors degrade long-horizon performance 5 . Figure 1 : Cube Bench flow: Given depth and seed, a VirtualCube state yields (i) a rendered image, (ii) a canonical text state, and (iii) a four-option set (A–D). The MLLM consumes the modalities and prompt, and returns output: Yes|No (Verification), A|B|C|D (Move Prediction / Closed-Loop), or DECREASE|NO_CHANGE|INCREASE (Causal Move-Effect). Existing visual and spatial reasoning benchmarks mostly operate in this static regime. Synthetic datasets such as CLEVR, GQA, and related visual question answering benchmarks probe fine-grained spatial relations, counting, and logical composition from a single image, but typically stop at one-shot answers about a fixed scene. More recent multimodal leaderboards continue this pattern, emphasizing captioning, OCR, and short-form VQA on web images 13 . These settings test local spatial understanding, yet they rarely couple perception to an explicit dynamics model, track progress toward a goal, or require models to maintain a consistent internal state over multiple actions. On the other side, rich agentic environments on the web or in 3D scenes 31 ; 10 introduce realistic noise and partial observability, but success metrics are often coarse, and it is difficult to disentangle failures of perception, planning, and control. We therefore lack a compact, generator-based domain that is simultaneously spatial and sequential, fully observed, exactly solvable, and regenerable on demand with strict fairness controls and oracle progress signals, making it possible to localize failures within a decision loop rather than aggregate them into a single static score. We introduce Cube Bench , a compact and fully controllable generator-based benchmark for studying long-horizon decision making under complete state information (see Figure 1 ). Conceptually, Cube Bench instantiates a simple TEA loop i.e., Transition \rightarrow Evaluation \rightarrow Argmin (Action): the cube dynamics define an exact transition model, oracle distance provides a scalar evaluation of each state, and the model’s role is to choose actions that (ideally) minimize distance to the goal at each step. Cube Bench uses the Rubik’s Cube as a testbed: it is visually simple, easy to simulate exactly, and combinatorially rich, with known optimal and near-optimal solvers that tell us precisely how many moves remain to reach the goal 11 ; 21 . This lets us, unlike static image corpora, generate unlimited episodes, render both an image and a concise text description of each state, present a small menu of candidate moves, and compute exact feedback on how much each move improves or worsens the situation. On top of this testbed we define seven tests that target different stages of the see \rightarrow evaluate \rightarrow act \rightarrow reflect \rightarrow recover loop: (i) reconstructing the full state from vision; (ii) checking consistency between vision and text; (iii) choosing the next move from a short list; (iv) judging whether a move helps or harms progress; (v) executing a sequence of moves to reach the goal; (vi) using self-reflection to revise an earlier choice; and (vii) trying to recover after the first error. Our framework enforces strict fairness controls, yielding measurements that are easy to replicate, difficult to game, and tailored to disentangle which stage of the TEA loop is failing. Evaluating strong open- and closed-source MLLMs under identical rules reveals a consistent picture. First, seeing is not reasoning : models that parse cube faces well still struggle to sustain improvement over multiple steps. Second, feedback breaks coherence : when outputs are fed back as inputs, error rates rise sharply with scramble depth, and recovery after a wrong move is rare. Third, reflection helps but is fragile : structured, label-aware reflection can repair some mistakes, yet unconstrained reflection can induce overthinking and instability 23 ; 16 ; 30 . Together, these results pinpoint lingering weaknesses in sequential spatial reasoning, i.e., state tracking and action evaluation, that are not revealed by current single-step benchmarks or coarse web-agent success rates. 2 Related Work Static perception benchmarks for MLLMs. A large body of work evaluates multimodal LLMs on single-shot perception tasks such as classification, detection, captioning, VQA, OCR, and charts, e.g., ImageNet, COCO, CLEVR, GQA, TextCaps, and ChartQA 22 ; 14 ; 8 ; 7 ; 24 ; 17 . Recent surveys synthesize these trends and note rapid leaderboard gains driven by scaling and instruction tuning 13 . Classic perception datasets (e.g., VQAv2) emphasize recognition under short contexts and have pushed models toward improved visual grounding and bias control 5 ; ScienceQA further probes multimodal reasoning with thought chains 15 . However, these settings rarely require closed-loop decision making, where actions alter the state and errors can compound across steps. Our benchmark targets this gap by measuring whether models can sustain improvements over multiple actions, given precise state feedback. Interactive / agent benchmarks. Web-scale agent environments assess language-to-action competence in long-horizon tasks (navigation, form-filling, retrieval, tool use) 31 ; 10 . While such benchmarks capture realistic challenges (dynamic pages, delayed rewards, partial observability), they introduce confounds, non-deterministic layouts, fragile tool chains, and evaluation noise, that complicate causal analysis of failures. We instead adopt a compact, fully controllable domain with deterministic transitions and exact progress signals, allowing us to isolate where breakdowns arise (perception, evaluation, selection, or reflection) without web-induced variance. Reasoning and

Read the complete text