DrawingBench: Evaluating Spatial Reasoning and UI Interaction Capabilities of Large Language Models through Mouse-Based Drawing Tasks
Hyunjun Kim, Sooyoung Ryu · 2025
Read DrawingBench: Evaluating Spatial Reasoning and UI Interaction Capabilities of Large Language Models through Mouse-Based Drawing Tasks by Hyunjun Kim, Sooyoung Ryu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-01174.txt.
Introduction
Hyunjun Kim and Sooyoung Ryu present DrawingBench in a 2025 paper on evaluating AI agents. As agentic systems act more autonomously, trust needs verifiable evaluation, proof that an agent behaves reliably. Existing benchmarks often lack the transparency and auditability to supply that proof.
DrawingBench evaluates agentic language models through mouse-based drawing tasks. The model must produce sequences of low-level GUI actions, moving and clicking a mouse to draw. Those tasks test spatial reasoning and interface control together, and every action can be checked. A verifiable drawing test gives auditors a concrete record of what the agent actually did.
Catalog record
| CREATOR | Hyunjun Kim, Sooyoung Ryu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-01174
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DrawingBench: Evaluating Spatial Reasoning and UI Interaction Capabilities of Large Language Models through Mouse-Based Drawing Tasks Hyunjun Kim, Sooyoung Ryu Abstract As agentic AI systems increasingly operate autonomously, establishing trust through verifiable evaluation becomes critical. Yet existing benchmarks lack the transparency and auditability needed to assess whether agents behave reliably. We present DrawingBench , a verification framework for evaluating the trustworthiness of agentic LLMs through spatial reasoning tasks that require generating sequences of low-level GUI actions. Unlike opaque evaluations, DrawingBench provides transparent, rule-based assessment: 8 objective criteria enable reproducible scoring, while action-level inspection allows stakeholders to audit agent behavior. Our framework comprises 250 diverse prompts across 20 categories and 4 difficulty levels, deterministic evaluation metrics, and an external oversight mechanism through multi-turn feedback that enables human control over agent refinement. Evaluating four state-of-the-art LLMs (Claude-4 Sonnet, GPT-4.1, GPT-4.1-mini, Gemini-2.5 Flash) across 1,000 tests, we establish both capabilities and limitations: models achieved 92.8% perfect performance with structured external feedback driving significant improvements (average +3.2%, up to +32.8% for complex scenes), but systematic error patterns emerged in tool state management and long-horizon planning. Notably, specification clarity proved more important than task complexity—models achieved 100% perfect performance when given explicit, verifiable criteria. These findings demonstrate that transparent evaluation frameworks can establish trust in agentic systems, with external oversight proving more reliable than self-correction for guiding agent behavior. Our open-source framework provides a template for trustworthy agent assessment. Code and data: https://github.com/hyunjun1121/DrawingBench * * footnotetext: Equal contribution. Introduction Spatial reasoning is a fundamental aspect of intelligence, enabling both humans and AI agents to understand and interact with their environments. In the context of large language models (LLMs), spatial reasoning capability is increasingly important as we move towards embodied or agentic AI systems that can manipulate the physical or digital world through language instructions. For instance, an AI assistant might be asked to arrange objects in a room or navigate a user interface (UI)—tasks that require understanding of spatial relations (above, below, inside, etc.) and geometry. Despite impressive advancements in LLMs’ general reasoning, their ability to perform precise spatial reasoning remains underexplored and poorly evaluated. Traditional evaluation benchmarks tend to focus on textual question-answering or reasoning puzzles, which may not capture an agent’s performance in interactive spatial tasks ( 11 ; 18 ) . For instance, a recent comprehensive evaluation of frontier models across many tasks, dubbed Humanity’s Last Exam , found significant gaps in LLM performance and calibration ( 16 ) , highlighting the need for targeted benchmarks that probe specific skills like spatial reasoning. Limitations of Existing Evaluations. Most LLM evaluation benchmarks suffer from three limitations for spatial and interactive tasks. First , they focus on text-based QA or classification, not action sequences. CLEVR ( 8 ) , RAVEN ( 19 ) , and SpartQA ( 15 ) test spatial reasoning via static questions, not action execution. Second , benchmarks lack fine-grained UI interaction. WebArena ( 20 ) and MiniWoB++ ( 12 ) use high-level actions (clicking links), not precise coordinate-level control. Third , multi-turn learning capability is largely untested. Most benchmarks use single-turn tasks, failing to probe iterative improvement from feedback. Our Solution. We introduce DrawingBench , a benchmark evaluating LLMs on spatial reasoning through drawing tasks on a canvas UI. Each task is specified by a natural language prompt (e.g., ”Draw a red triangle inside a blue square” ), and the LLM outputs a sequence of low-level GUI actions (mouse movements, clicks, tool selections). This approach differs from prior benchmarks: the model constructs spatial configurations via actions rather than answering questions. Our benchmark tests spatial understanding, tool manipulation, and sequential planning simultaneously. Crucially, evaluation is text-only—the LLM relies on internal reasoning to plan drawings without seeing images. Trust through Verifiable Evaluation. As agentic AI systems increasingly operate autonomously, establishing trust becomes paramount. DrawingBench embodies three essential properties for trustworthy evaluation: (1) Transparency —eight objective criteria enable stakeholders to understand performance measurement; (2) Auditability —deterministic scoring allows reproduction through action sequence inspection; (3) External oversight —multi-turn feedback provides human control over agent refinement. Unlike opaque metrics or self-assessment, our rule-based system eliminates subjective judgment and enables action-level verification, aligning with best practices for deploying trustworthy agents. DrawingBench comprises 250 diverse tasks spanning 20 categories and four difficulty levels. Our automated evaluation system executes action sequences and measures performance against eight objective criteria. The framework supports multi-turn interaction: models receive structured feedback and attempt corrections. Contributions. We contribute: (1) DrawingBench —first benchmark integrating spatial reasoning and GUI action execution with 250 prompts; (2) Automated evaluation —eight quantitative criteria, four error types, reproducible scoring; (3) Multi-turn protocol —tests self-correction from explicit feedback ( 14 ; 17 ) ; (4) Empirical study —1000 trials with four state-of-the-art LLMs analyzing performance across difficulty levels; (5) Open-source release ---all code and data. 1 1 1 https://github.com/hyunjun1121/DrawingBench Key Findings. Models achieved 92.8% perfect scores, with structured feedback yielding +3.2% average improvement ( +32.8% for complex scenes). Three insights emerged: (1) Text-based spatial reasoning is highly effective—models executed complex 15+ action sequences without visual perception. (2) Difficulty paradox : “Hard” tasks achieved 100% perfect performance versus “Medium” (92.8%)—specification clarity matters more than complexity. (3) Structured feedback proves effective, but “Very Hard” tasks requiring long-horizon planning remained challenging (60% success). Section 2 reviews related work. Section 3 details methodology. Section 4 presents results. Section 5 discusses implications and limitations. Section 6 concludes. Related Work Spatial Reasoning Benchmarks Evaluating spatial reasoning has been a focus across vision and language domains. In vision+language, CLEVR ( 8 ) presents synthetic 3D scenes with compositional reasoning questions, while RAVEN ( 19 ) tests abstract visual pattern completion. In pure text, SpartQA ( 15 ) assesses understanding of spatial relations through paragraph descriptions and questions. Table 1 summarizes some representative spatial reasoning benchmarks. Most are static in nature: the model reads or observes a scenario and produces an answer (classification or text). They typically measure reasoning capability via accuracy on predefined questions. However, none of these require the model to generate a sequence of actions to physically realize a spatial configuration. In contrast, DrawingBench is dynamic : the model must “draw” the answer, not just tell it. This calls for integrating spatial reasoning with a decision-making process to carry out the drawing actions. Benchmark Modality Type Output CLEVR Vision Static QA SpartQA Text Static QA RAVEN Vision Static Choice DrawingBench Text Dynamic Actions Table 1: Comparison of spatial reasoning benchmarks. DrawingBench uniquely requires dynamic action generation rather than static question answering, integrating spatial understanding with executable UI interactions. Program Synthesis and Visual Generation Our work sits at the intersection of program synthesis and visual generation. Code synthesis benchmarks like HumanEval ( 3 ) , MBPP ( 1 ) , and APPS ( 7 ) evaluate algorithmic correctness but not spatial outputs. Visual generation work like Sketch-RNN ( 6 ) and visual program synthesis ( 5 ) demonstrates that graphics can be represented as executable sequences, but these train on data rather than evaluate zero-shot spatial reasoning. DrawingBench bridges these areas by requiring action sequences that are both syntactically valid and geometrically correct. User Interface Interaction Benchmarks UI interaction benchmarks evaluate agents on GUI and web tasks. MiniWoB++ ( 12 ) tests basic web form interactions