TurtleBench: A Visual Programming Benchmark in Turtle Geometry

Sina Rismanchian, Yasaman Razeghi, Sameer Singh, Shayan Doroudi · 2024

Read TurtleBench: A Visual Programming Benchmark in Turtle Geometry by Sina Rismanchian, Yasaman Razeghi, Sameer Singh, Shayan Doroudi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00264.txt.

Description

Abstract Humans have the ability to reason about geometric patterns in images and scenes from a young age. However, developing large multimodal models (LMMs) capable of similar reasoning remains a challenge, highlighting the need for robust evaluation methods to assess these capabilities.

Catalog record

CREATORSina Rismanchian, Yasaman Razeghi, Sameer Singh, Shayan Doroudi
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-00264

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TurtleBench: A Visual Programming Benchmark in Turtle Geometry Sina Rismanchian, Yasaman Razeghi, Sameer Singh, Shayan Doroudi Abstract Humans have the ability to reason about geometric patterns in images and scenes from a young age. However, developing large multimodal models (LMMs) capable of similar reasoning remains a challenge, highlighting the need for robust evaluation methods to assess these capabilities. We introduce TurtleBench, a benchmark designed to evaluate LMMs’ capacity to interpret geometric patterns—given visual examples, textual instructions, or both—and generate precise code outputs. Inspired by turtle geometry, a notion used to teach children foundational coding and geometric concepts, TurtleBench features tasks with patterned shapes that have underlying algorithmic logic. Our evaluation reveals that leading LMMs struggle significantly with these tasks, with GPT-4o achieving only 19% accuracy on the simplest tasks and few-shot prompting only marginally improves their performance ( < 2 % <2\% ). TurtleBench highlights the gap between human and AI performance in intuitive and visual geometrical understanding, setting the stage for future research in this area. TurtleBench stands as one of the few benchmarks to evaluate the integration of visual understanding and code generation capabilities in LMMs, setting the stage for future research. Code and Dataset for this paper is provided here: https://github.com/sinaris76/TurtleBench 1 Introduction Large Multimodal Models (LMMs) have the potential to handle tasks that combine visual, linguistic, and reasoning abilities, previously achievable only by humans. Indeed, LMMs such as GPT4-V ( Yang et al., 2023 ) and Gemini 1.5 flash ( Team et al., 2023b ; Fu et al., 2023 ) have been shown to be state-of-the-art models in solving multi-modal tasks such as visual question answering ( Goyal et al., 2017 ; Liu et al., 2024 ) , coding for visual multimodal questions ( Li et al., 2024a ) , visual mathematical questions ( Lu et al., 2023 ) , chart question answering ( Masry et al., 2022 ) , etc. Despite these successes, there remains the question of how LMMs perform in tasks that intertwine visual reasoning and programming knowledge. That is, given an image of a geometric pattern (and/or a verbal description of the pattern) can LMMs generate code that would be able to procedurally generate that pattern? Indeed, Bubeck et al. (2023) showed that the large language model (LLM) with no visual training data was able to create a unicorn in TikZ—the LaTeX-based graphics drawing library. This feat amazed many and provoked many discussions on the intelligence of large language models—but how general is this ability? Figure 1: An illustration of existing types and modes in TurtleBench. In this work, we introduce TurtleBench, a set of manually crafted image/text to code tasks in turtle geometry ( Papert, 1972 ; Abelson and diSessa, 1986 ) to evaluate the abilities of these models to combine visual pattern recognition, mathematical reasoning, Python programming, and abstract geometrical reasoning. To ensure the visual inputs and the programming language remain straightforward, TurtleBench harnesses turtle geometry, a concept widely recognized for its effectiveness in introducing programming concepts to children within the K-12 education system. In turtle geometry, a turtle acts as a programmable object that navigates the screen, drawing as it goes and turning at specified angles, to create simple visual patterns. The primary objective within this framework is to generate code capable of producing simple visual inputs. These visual inputs consist of basic geometric shapes, and the programming syntax required is intentionally limited and straightforward. An example of such a task is presented in the left side of Figure 1 . As illustrated, the input image is the shape of a simple square and the corresponding code only uses two simple turtle functions ( forward and right ) along with a simple for loop. This simplicity makes TurtleBench an effective benchmark for evaluating the capabilities of Large Multimodal Models (LMMs). To reflect different real-world use cases of an LMM in the domain of Turtle and also cover the broad range of underlying reasoning abilities, TurtleBench includes 260 tasks with a variety of types and modalities. The different types and input/output modalities are presented in Figure 1 . We define two types of tasks scratch and tweak in TurtleBench. Scratch tasks challenge models to generate Python code for a specified shape using the Turtle library based on inputs that could be an image of the shape, a textual description, or both. This subset evaluates model’s proficiency in recognizing patterns within the shape and accurately translating these into executable code. Conversely, tweak tasks are designed to probe deeper into a model’s understanding, examining its ability to comprehend the implications of described modifications to shapes—such as connecting midpoints (example in Figure 1 )—and their representation in the image. Here, models are provided with a base shape and are instructed to create the desired alteration in shape. Instructions for these modifications may be provided visually or textually . To simplify, in a subset of the tweak tasks, we adopt a code editing approach, supplying the original shape’s code and directing the model to edit this code to generate the target shape. We conduct an evaluation of leading LMMs on TurtleBench code generation and code editing tasks, utilizing zero-shot and visual chain-of-thought approach ( Singh et al., 2023 ) across text-only, image-only, and mixed (text and image) input modalities. Our findings reveal that these models generally perform poorly across all setups and variety of tasks and modalities. Both GPT-4o and Gemini 1.5 Flash struggle with TurtleBench tasks, failing to solve more than 75% of them. Our results show that performance improves when tasks are presented as text rather than images, suggesting that integrating visual and linguistic information, especially for visual pattern recognition, requires further refinement. When tested with a custom library mimicking Python Turtle but using different command names, models showed a significant performance drop, revealing difficulties in generalizing visual reasoning to unfamiliar syntax. Even when allowed to choose their own programming language, models consistently failed to generate correct code, indicating broader challenges in translating visual instructions into functional programming outputs. These findings show that our benchmark poses a significant challenge for LMMs, offering key insights into their limitations. Our evaluation highlights gaps in integrating visual reasoning with programming and raises important questions for future research to address. 2 Overview of TurtleBench In turtle geometry ( Abelson and diSessa, 1986 ) , a turtle is a programmable object on the screen that leaves a trace while moving. As illustrated in Figure 1 left side, we see an example of how creating a simple geometric shape, like a square, involves the turtle moving forward and executing turns four times. It is a powerful, intuitive tool that enables novice learners to start learning programming by creating aesthetically beautiful artifacts. Although Turtle programming nowadays is used more as a tool to foster computational thinking, it can be used to teach geometry and mathematical reasoning Marji (2014) ; Clements and Sarama (2013) as it enables learners to explore and learn geometrical relationships between shapes. The intuitive nature and learnability of turtle geometry, along with its ability to generate patterns of diverse complexities, make it a compelling concept upon which to base a benchmark for LMMs. In the following, we describe our benchmark in detail. 2.1 TurtleBench Task Types

Read the complete text