ShapeCodeBench : A Renewable Benchmark for Perception-to-Program Reconstruction of Synthetic Shape Scenes
Shivam Kumar · 2026
Read ShapeCodeBench : A Renewable Benchmark for Perception-to-Program Reconstruction of Synthetic Shape Scenes by Shivam Kumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11680.txt.
Introduction
Shivam Kumar introduces ShapeCodeBench, a renewable benchmark for turning perception into programs. Given a rendered image, a model must emit a drawing program that an evaluator re-renders and checks against the target. The version-one language uses four primitives on a 512 canvas, and every instance comes from a seeded generator, so fresh sets can cut contamination. A frozen split holds 150 samples across three tiers, scored by exact match, pixel accuracy and foreground IoU. The strongest configuration preserves much foreground structure yet misses exact match on small errors.
Catalog record
| CREATOR | Shivam Kumar |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29358 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.11680
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ShapeCodeBench : A Renewable Benchmark for Perception-to-Program Reconstruction of Synthetic Shape Scenes Thanks: Code, benchmark data, and figures: https://github.com/shivamk3r/shape-code-bench . Archived release DOI: 10.5281/zenodo.20132286 . Shivam Kumar Abstract We introduce ShapeCodeBench , a synthetic benchmark for perception-to-program reconstruction : given a rendered raster, a model must emit an executable drawing program that a deterministic evaluator re-renders and compares. The DSL has four primitives on a 512 × 512 512\times 512 black-on-white canvas, but every instance is generated from a seeded RNG, so fresh held-out sets can be minted to mitigate benchmark contamination. Because both instance generation and scoring are automatic, the same loop can refresh evaluations quickly without per-instance human annotation or manual judging. We release a frozen split, eval_v1 ( 150 150 samples, 50 50 per difficulty tier), scored by exact match, pixel accuracy, and foreground IoU alongside parse and execution success. Evaluating four reasoning-effort configurations of two frontier multimodal models – Claude Opus 4.7 (1M context) at high and max effort, and GPT-5.5 at medium and extra_high reasoning effort – against an empty-program floor and a classical-CV heuristic baseline exposes a tier-dependent crossover: the heuristic leads easy-tier exact match ( 0.26 0.26 vs. at most 0.08 0.08 for any multimodal configuration) by individuating separated connected components, but collapses on medium and hard scenes as overlapping shapes fuse; the strongest multimodal model by foreground IoU ( GPT-5.5 at extra_high effort) retains most of the spatial structure and leads foreground IoU on every tier (up to 0.87 0.87 ), yet misses exact match by small parameter errors, while Claude Opus 4.7 (1M) trails the heuristic on foreground IoU at both effort levels. Best overall exact match is 0.087 0.087 (heuristic) and 0.027 0.027 among multimodal models, so ShapeCodeBench is far from saturated. Benchmark code, frozen dataset, and full run artifacts are released to support independent replication and extension. 1 Introduction Modern multimodal models are increasingly evaluated on their ability to turn images into code. Work on screenshot-to-HTML [ 2 , 16 ] , structure extraction from webpages, LaTeX, and music scores [ 14 ] , and symbolic vector generation [ 10 , 19 ] all ask the same underlying question in different clothes: can a model look at a picture and produce the program that generated it? Earlier work on visual program induction [ 15 , 12 , 9 , 8 , 4 , 5 ] framed this problem as inverse graphics over a small symbolic DSL, and more recently TurtleBench [ 13 ] has revived it as a benchmark-first target for large vision-language models. Across these lines of work, three design pressures keep recurring: (1) deterministic execution so that scoring is principled; (2) render-based scoring so that semantically equivalent programs are not penalized for textual differences [ 14 ] ; and (3) controlled generation so that failure modes can be attributed to specific visual factors, in the tradition of CLEVR [ 6 , 1 ] . Most existing benchmarks satisfy one or two of these, but few satisfy all three while remaining renewable – that is, cheap enough to regenerate when an existing split becomes contaminated. For model development, renewability also changes the feedback cycle: a researcher can generate fresh instances, run a model, and obtain objective scores without commissioning new labels or manual judgments for each refreshed example. We present ShapeCodeBench , a perception-to-program benchmark that attempts to hit the intersection of these pressures. The task is narrow by design: the DSL has exactly four primitives ( filled_circle , circle , filled_square , square ) and the canvas is fixed at 512{\times}512 grayscale with a black foreground on a white background. Every sample is generated from a seeded RNG with explicit difficulty controls on shape count, size, stroke width, overlap, and canvas clipping. The evaluator parses predictions with a safe restricted Python parser, re-renders them through the same deterministic Pillow pipeline used to produce the target, and compares rasters. Contributions. We make the following contributions: 1. We release the ShapeCodeBench benchmark: a four-primitive drawing DSL, a safe restricted parser, a seeded scene generator with three difficulty tiers, and a render-based evaluator with five primary metrics. 2. We freeze an evaluation split, eval_v1 , of 150 samples with deterministic seeds, and publish per-sample raster hashes to make the exact evaluation instances reproducible across platforms. 3. We make benchmark refresh a first-class workflow: new held-out splits can be generated from fresh seeds and scored automatically, enabling fast regression-style feedback while avoiding per-instance human annotation or manual judging. This mitigates exact-instance contamination but does not prevent models from learning the generator distribution. 4. We release a provider-agnostic runner that records prompts, model configuration, raw outputs, normalized predictions, metrics, and per-sample artifacts, making model evaluations auditable and easy to extend. 5. We report baseline results for four multimodal model configurations – Claude Opus 4.7 (1M context) at high and max effort, and GPT-5.5 at medium and extra_high reasoning effort – alongside two non-LLM baselines (empty program, classical-CV heuristic). The strongest multimodal model by foreground IoU ( GPT-5.5 / extra_high ) reaches mean foreground IoU 0.87 , the best multimodal exact match remains 0.027 , and the classical heuristic leads easy-tier exact match at 0.26 – a tier-dependent crossover that confirms ShapeCodeBench is not saturated and exposes distinct failure modes in perception versus structured code emission. Effort tier helps modestly ( max > high for Claude on FG-IoU; extra_high > medium for GPT-5.5) but does not close either gap. The rest of the paper is organized as follows. Section 2 places ShapeCodeBench against prior work. Section 3 describes the DSL, generator, renderer, and evaluator. Section 4 details the experimental setup and headline results. Section 5 analyzes failure modes, the heuristic-vs-LLM gap, and difficulty validity. Section 6 discusses limitations and future work. 2 Related Work ShapeCodeBench sits at the intersection of three established lines of work: visual program induction, synthetic diagnostic benchmarks, and image-to-code evaluation of multimodal models. Visual program induction and inverse graphics. Predicting executable programs from images has a long history. CSGNet [ 15 ] infers constructive solid geometry programs from 2D and 3D shapes, and is the direct conceptual ancestor of ShapeCodeBench . [ 12 ] learn to describe scenes with a DSL that supports loops and grouping, demonstrating that compositional program structure – and not only local shape identity – can be recovered from a single image. [ 9 ] and [ 8 ] extend program induction to perspective scenes and repeated 3D structure, while [ 4 ] studies parametric primitives with explicit function correlations, close in spirit to the integer-parameter DSL of ShapeCodeBench . LILO [ 5 ] synthesizes reusable program libraries across domains including graphics composition, providing a template for symbolic baselines. These systems prove that the image-to-program problem is well-defined; none of them are benchmark-first evaluations of modern multimodal models. Benchmark design. CLEVR [ 6 ] showed how much scientific value a synthetic, carefully factorized benchmark can add when it is designed to reduce spurious shortcuts and expose reasoning failure modes. CLOSURE [ 1 ] extended this insight by probing systematic generalization. We adopt the same diagnostic posture: rather than building a large, noisy dataset, we expose the axes of variation explicitly and allow researchers to regenerate