ManiBench: A Benchmark for Testing Visual-Logic Drift and Syntactic Hallucinations in Manim Code Generation
Nabin Oli · 2026
Read ManiBench: A Benchmark for Testing Visual-Logic Drift and Syntactic Hallucinations in Manim Code Generation by Nabin Oli (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13251.txt.
Introduction
Nabin Oli introduces ManiBench, a benchmark for Manim code generation by language models. Code that must produce dynamic teaching visuals needs more than plain logic and syntax checks. ManiBench targets syntactic hallucinations, where valid Python calls Manim APIs that do not exist or are deprecated. It also targets visual-logic drift, where visuals diverge from the intended math through timing errors. The set spans five difficulty levels across calculus, linear algebra, probability, topology, and AI. Scoring covers executability, version conflicts, alignment, and coverage.
Catalog record
| CREATOR | Nabin Oli |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32278 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.13251
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ManiBench: A Benchmark for Testing Visual-Logic Drift and Syntactic Hallucinations in Manim Code Generation Nabin Oli Abstract Traditional benchmarks like HumanEval and MBPP test logic and syntax effectively, but fail when code must produce dynamic, pedagogical visuals. We introduce ManiBench , a specialized benchmark evaluating LLM performance in generating Manim CE code, where temporal fidelity and version-aware API correctness are critical. ManiBench targets two key failure modes: Syntactic Hallucinations (valid Python referencing non-existent or deprecated Manim APIs) and Visual-Logic Drift (generated visuals diverging from intended mathematical logic through timing errors or missing causal relationships). The benchmark comprises 12 problems in pilot phase across five difficulty levels spanning calculus, linear algebra, probability, topology, and AI, grounded in analysis of 3Blue1Brown’s ManimGL source ( 53,000 lines, 143 scene classes). Evaluation uses a four-tier framework measuring Executability, Version-Conflict Error Rate, Alignment Score, and Coverage Score. An open-source framework automates evaluation across multiple models and prompting strategies. Code, data and benchmark suite are available at https://github.com/nabin2004/ManiBench . and the dataset is hosted on https://huggingface.co/datasets/nabin2004/ManiBench . keywords Syntactic Hallucinations, Visual-Logic Drift, Manim CE, Code Generation, Benchmarking 1 Introduction The rise of large language models (LLMs) has accelerated research in code generation. Benchmarks such as HumanEval [ 2 ] , MBPP [ 1 ] , APPS [ 5 ] and SWE-Benchmarks [ 9 ] were commonly used to evaluate LLM coding ability. These benchmarks emphasize logical correctness, syntactic validity, and output matching i.e., whether generated code solves the task, executes without errors, and yields the expected results. While well suited for algorithmic problems, these criteria are insufficient for domains in which code produces continuous, time-dependent visual outputs. Manim, a Python animation engine developed by Grant Sanderson (3Blue1Brown), constructs mathematical animations by composing scene objects, applying geometric and visual transformations, and controlling timing. A Manim script may be syntactically valid yet fail to convey the intended concept; we identify three common failure modes: • Incorrect visual semantics: animations move in unintended directions or transform the wrong objects. • Timing misalignments: events occur out of order or at inappropriate times, breaking causal or pedagogical flow. • Pedagogical failure: animations obscure rather than clarify the concept (missing annotations, poor sequencing, or absent explanatory cues). Manim exists in major two variants. Manim CE (Community Edition) is an actively maintained, open-source implementation with a modern API, whereas Manim GL (the original 3B1B fork) contains deprecated constructs and hand-optimized code paths. LLMs frequently conflate APIs across these variants or reference renamed/moved functions, producing code that fails under particular library versions. 2 Contributions ManiBench makes four key contributions: 1. Visual-Logic Metrics. We define an Alignment Score and a four-dimensional Coverage Score to evaluate the generated animations pedagogical intent and syntactic validity. 2. Version-Aware Evaluation. We explicitly test version-conflict errors and deprecated API usage, with 145 documented GL \to CE incompatibilities across eight categories, measuring whether code adheres to a specific Manim version’s API contract. 3. Curated Pilot Dataset with Reference Code Analysis. We provide 12 hand-crafted benchmark problems drawn from 3Blue1Brown’s published videos, backed by comprehensive analysis of {\sim}53{,}000 lines of original source code including 143 scene classes, 120 visual techniques, and detailed visual-event specifications. 4. Automated Evaluation Framework. We release an evaluation pipeline supporting six LLMs across five prompting strategies, with automated metric computation for executability, version conflicts, alignment, and coverage. 3 Background Evaluation of large language models (LLMs) for program synthesis has progressed from function-level correctness to repository-scale reasoning. Early benchmarks such as HumanEval [ 2 ] formalized unit-test-based validation through the Pass@k metric, establishing executability as the primary indicator of correctness. MBPP [ 1 ] extended this setting to natural-language-to-code tasks across entry-level Python problems. More recent benchmarks such as DS-1000 [ 7 ] incorporated third-party library usage, while SWE-Bench introduced repository-scale issue resolution requiring multi-file contextual reasoning [ 9 ] . Benchmark Metric Focus / Area HumanEval / MBPP [ 10 ] Pass@k Python program syn- thesis, algorithmic tasks SWE-Bench [ 9 ] Patch rate Software engineering / repo-level edits DSCodeBench [ 6 ] API cov. Data-science APIs and pipelines Code2Video [ 4 ] Aesthetic Educational video/ storyboarding SVGenius [ 3 ] Visual SVG editing / static visual generation Table 1: Related benchmarks and limitations addressed by ManiBench. Despite increasing scale and realism, these benchmarks remain fundamentally test-driven. This evaluation paradigm does not account for renderable behavior, temporal sequencing, or alignment between code structure and intended visual semantics. 3.1 Visual-Logic Drift in Executable Code Visual-logic drift in the context of Manim code generation can be defined as the divergence between the intended mathematical narrative and the rendered animation, even when code executes successfully [ 8 ] . Unlike traditional logical errors, drift may arise from subtle structural mismatches: incorrect object transformations, missing intermediate states, improper sequencing of animation calls, or violations of causal dependencies across scenes. Animation scripting introduces temporal constraints and multi-step causal structure, requiring preservation of event ordering and pedagogical coherence. Current benchmarks do not quantify this form of semantic drift in executable animation environments. 3.2 Syntactic Hallucinations and Version Conflicts Hallucination in code generation has been widely characterized as the production of syntactically valid but semantically incorrect outputs, including fabricated APIs, deprecated function usage, and identifier conflicts [ 11 ] . Prior work demonstrates that hallucination rates increase in low-frequency or evolving niche libraries, and that such errors often evade static analysis because they remain grammatically valid Python [ 11 ] . In rapidly evolving libraries, version misalignment introduces an additional layer of failure. Models may generate code consistent with legacy documentation while the prompt implicitly assumes a newer API specification. These errors are particularly difficult to detect in animation frameworks, where partial execution may mask deeper incompatibilities. 3.3 Positioning of ManiBench ManiBench is situated at the intersection of code-generation benchmarking, hallucination analysis, and visual-temporal reasoning. While prior benchmarks evaluate executability and repository-scale edits, and hallucination studies analyze API-level inconsistencies, none jointly assess version-aware API correctness and temporal visual alignment in mathematical animation scripting. ManiBench addresses this gap through drift-sensitive task design and a multi-tier evaluation framework that extends beyond unit-test validation to measure semantic fidelity in renderable educational code. 4 Problem Definition We deal with two distinct failure modes that traditional code-generation benchmarks fail to capture when evaluating dynamic visual outputs: Syntactic Hallucinations and Visual-Logic Drift . 4.1 Syntactic Hallucinations Syntactic hallucinations occur when an LLM produces code that is grammatically valid Python but semantically invalid within the target Manim CE environment. This failure mode typically manifests when the model: • Invokes non-existent classes : e.g., inventing MCircle instead of utilizing the standard Circle class; • Employs deprecated methods : e.g., relying on outdated API calls like mobject.scale() rather than the modern mobject.scale_to_fit_width() ; • Generates incorrect method signatures : passing unsupported arguments to valid functions; • Conflates API versions : mixing Manim GL syntax with Manim CE, such as utilizing OpenGL-specific rendering commands that crash the CE pipeline. ⬇ # HALLUCINATION: ’MCircle’ does not exist circle = MCircle ( color = BLUE ) # Expected: Circle # HALLUCINATION: Deprecated method usage circle . apply_matrix ([[1, 0], [0, 1]]) Listing 1: Examples of syntactic hallucinations. 4.2 Visual-Logic Drift Visual-logic drift emerges when the generated code executes without runtime exceptions but fails to accurately represent the