Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models
Davie Chen · 2026
Read Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models by Davie Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-28406.txt.
Introduction
Davie Chen, writing alone, asks whether image generators can draw science. Researchers increasingly ask text-to-image models for mechanism diagrams, experiment schematics, and graphical abstracts. Yet existing benchmarks grade natural photos for realism and object counting. They do not grade whether a figure's labels are right or its arrows mean anything.
Chen introduces SciDraw-Bench, thirty-two figure tasks across eight figure types and ten fields, each with machine-checkable rules for labels, relations, components, conventions, and forbidden elements. Scoring covers text fidelity, semantic correctness, structural quality, and fit with field conventions. In a pilot across all figure types, a system built for the domain beat general text-to-image models on every dimension, with the widest gaps in semantics and conventions. Text fidelity stayed hardest for all models.
Catalog record
| CREATOR | Davie Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.28406
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can AI Draw Science? A Benchmark for Evaluating Scientific Figure Generation by Text-to-Image and Multimodal Models Davie Chen Abstract Text-to-image and multimodal generative models are increasingly used to produce scientific figures—mechanism diagrams, experimental-design schematics, conceptual frameworks, and graphical abstracts. Yet existing image-generation benchmarks (e.g., GenEval, T2I-CompBench, DPG-Bench) evaluate natural images and measure compositionality, object counting, or photorealism. None of them measure the properties that determine whether a generated scientific figure is usable: correct and legible text labels, faithful depiction of entities and their relations, coherent diagrammatic structure, and adherence to disciplinary drawing conventions. We introduce SciDraw-Bench , a benchmark of 32 structured scientific-figure generation tasks spanning eight figure types and ten disciplines, where each task pairs a natural-language prompt with a machine-checkable specification of required labels, relations, components, conventions, and negative constraints. We propose a four-dimensional evaluation protocol—Text Fidelity (OCR-based label recall and character error rate), Semantic Correctness (vision–language-model judging against the specification), Structural Quality, and Convention Adherence—together with a meta-evaluation protocol and a preliminary inter-judge reliability analysis (human-rating validation is ongoing). We evaluate a domain-specific system, SciDraw AI , against representative general-purpose text-to-image models, and outline a code-to-figure baseline (LLM-generated vector graphics) as a planned extension. The task specifications and evaluation harness are available from the author on request, supporting reproducible measurement of progress on scientific figure generation. In a pilot over all eight figure types, the domain-specific system substantially outperforms the general-purpose text-to-image baselines on every dimension and figure type, with the largest gaps on semantic correctness and disciplinary-convention adherence; text fidelity remains the hardest dimension for all systems. Keywords: text-to-image generation, scientific figures, benchmark, evaluation, vision–language models, reproducibility 1 Introduction Scientific figures are a primary medium of scholarly communication: mechanism diagrams, experimental-design schematics, conceptual frameworks, and graphical abstracts compress complex ideas into a form that supports peer review, replication, and teaching ( Tufte 2001 ; Rougier et al. 2014 ) . Producing them well has traditionally required substantial time and specialized tooling. The rapid improvement of text-to-image (T2I) diffusion models ( Rombach et al. 2022 ; Saharia et al. 2022 ) , autoregressive image models ( Yu et al. 2022 ) , and multimodal foundation models ( Betker et al. 2023 ; Team et al. 2023 ) has made it plausible that such figures could be generated directly from natural-language descriptions, and a number of systems now target this use case. Whether current models can actually draw science , however, is an open empirical question—and one that existing benchmarks do not answer. The dominant T2I evaluations measure properties of natural images: GenEval ( Ghosh et al. 2023 ) and T2I-CompBench ( Huang et al. 2023 ) probe object presence, counting, color binding, and spatial composition; DPG-Bench ( Hu et al. 2024 ) measures dense prompt following; HEIM ( Lee et al. 2023 ) aggregates aesthetics, alignment, robustness, and bias; and FID ( Heusel et al. 2017 ) and CLIPScore ( Hessel et al. 2021 ) score distributional realism and image–text alignment. None of these capture what makes a scientific figure usable : (i) Text fidelity. Scientific figures are label-dense. A mechanism diagram with garbled or misspelled gene names is worthless, yet T2I models are notoriously unreliable at in-image text ( Chen et al. 2023 ; Yang et al. 2023 ) . (ii) Semantic correctness. The right entities must appear in the right relations. “PTEN inhibits PIP3” must be drawn as inhibition, not activation. (iii) Structural quality. Reading order, arrow directionality, and grouping must communicate the intended logic. (iv) Convention adherence. Disciplines have visual grammars (membranes as bilayers, inhibition as blunt arrows, counter-current condenser flow) that domain readers expect. We address this gap with SciDraw-Bench . Our contributions are: (1) A benchmark dataset of 32 scientific-figure generation tasks across eight figure types and ten disciplines. Crucially, the dataset consists of tasks , not reference images: each task is a prompt paired with a machine-checkable specification (required labels, relations, components, conventions, negative constraints). There is no single ground-truth image, because a specification admits many correct renderings (Section 3 ). (2) An evaluation protocol tailored to scientific figures, combining OCR-based Text Fidelity, vision–language-model (VLM) judging of Semantic Correctness, and rubric-based Structural Quality and Convention Adherence (Section 4 ). (3) A meta-evaluation protocol for validating the automatic metrics against human expert ratings using rank correlation and inter-annotator agreement—the step that distinguishes a measurement instrument from an ad-hoc tool comparison—together with a preliminary inter-judge reliability analysis (Section 4 ); the human-rating study is ongoing. (4) A pilot comparative study of a domain-specific system against representative general-purpose text-to-image baselines, with a qualitative error taxonomy (Sections 5 – 8 ). The task specifications and evaluation harness are available from the author on request so that the benchmark can be re-run as models evolve. This manuscript fixes the benchmark and protocol and reports a pilot evaluation across all eight figure types (Sections 6 – 8 ); the human meta-evaluation is ongoing. 2 Related Work Text-to-image evaluation. Early evaluation of generative image models relied on distributional metrics such as FID ( Heusel et al. 2017 ) and on image–text alignment scores such as CLIPScore ( Hessel et al. 2021 ) . As prompt following became the bottleneck, prompt-centric benchmarks emerged: PartiPrompts ( Yu et al. 2022 ) , GenEval ( Ghosh et al. 2023 ) , T2I-CompBench ( Huang et al. 2023 ) , and DPG-Bench ( Hu et al. 2024 ) measure object presence, counting, attribute binding, spatial relations, and dense prompt adherence. HEIM ( Lee et al. 2023 ) broadens evaluation to multiple human-aligned dimensions. All of these target natural-image generation; none define a notion of diagrammatic correctness or measure in-image scientific text, which is the focus of SciDraw-Bench. Text rendering in images. The difficulty of rendering legible text has motivated specialized methods such as TextDiffuser ( Chen et al. 2023 ) and GlyphControl ( Yang et al. 2023 ) , and corresponding evaluations based on OCR accuracy. We adopt an OCR-based label-recall metric in the same spirit, but apply it to the domain-specific labels demanded by a figure specification rather than to free-form text rendering. Model- and VLM-as-judge. Using strong models to judge generated outputs is now standard for open-ended tasks ( Zheng et al. 2023 ) . We use a VLM-as-judge ensemble for the semantic and structural dimensions, force structured per-item verdicts against an explicit rubric, and report inter-judge agreement as a reliability check; validation against human ratings is part of our protocol and is ongoing. Scientific visualization and tools. Principles of effective scientific figures are long established ( Tufte 2001 ; Rougier et al. 2014 ) , and tools such as BioRender and domain-specific AI platforms ( SciDraw AI 2025 ) aim to lower the cost of producing them. Prior policy-oriented work has surveyed the editorial acceptability of AI-generated figures ( Chen 2026 ) ; here we instead ask the orthogonal, measurable question of how good current systems are at the task. 3 SciDraw-Bench 3.1 Design principle: tasks, not images A scientific figure specification is satisfied by many distinct images. A correct CAR-T mechanism diagram may place the T cell on the left or the right, use different iconography for the receptor, and still be fully correct. Pinning a single reference