PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning

Qiran Zhang † Yuheng Wang † Runde Yang Lin Wu Jingru Fan Shu Yao Jie Zhang Tianle Zhou Huatao Li Ruijie Shi Yihan Li Chen Qian 🖂 School of Artificial Intelligence, Shanghai Jiao Tong University {qrzhang_23,qianc}@sjtu.edu.cn Code : github.com/positionprivacy/PRISM Dataset : datasets/posprivacy/PRISM · 2026

Read PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning by Qiran Zhang † Yuheng Wang † Runde Yang Lin Wu Jingru Fan Shu Yao Jie Zhang Tianle Zhou Huatao Li Ruijie Shi Yihan Li Chen Qian 🖂 School of Artificial Intelligence, Shanghai Jiao Tong University {qrzhang_23,qianc}@sjtu.edu.cn Code : github.com/positionprivacy/PRISM Dataset : datasets/posprivacy/PRISM (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19382.txt.

Introduction

Zhang and colleagues build PRISM, a benchmark that tests whether language models can write code producing spatially correct animated videos. It holds 10,372 checked instruction-code pairs across 437 subjects in English and Chinese, far larger than earlier sets. A four-part funnel scores runnable code, layout accuracy over full animations, and two measures of motion richness. Tests of seven leading models expose a wide execution-spatial gap: code often runs yet lands objects wrongly, with pass rates falling about 41 percent.

Catalog record

CREATORQiran Zhang † Yuheng Wang † Runde Yang Lin Wu Jingru Fan Shu Yao Jie Zhang Tianle Zhou Huatao Li Ruijie Shi Yihan Li Chen Qian 🖂 School of Artificial Intelligence, Shanghai Jiao Tong University {qrzhang_23,qianc}@sjtu.edu.cn Code : github.com/positionprivacy/PRISM Dataset : datasets/posprivacy/PRISM
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33600
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19382

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

PRISM: A Benchmark for Programmatic Spatial-Temporal Reasoning Qiran Zhang † Yuheng Wang † Runde Yang Lin Wu Jingru Fan Shu Yao Jie Zhang Tianle Zhou Huatao Li Ruijie Shi Yihan Li Chen Qian 🖂 School of Artificial Intelligence, Shanghai Jiao Tong University {qrzhang_23,qianc}@sjtu.edu.cn Code : github.com/positionprivacy/PRISM Dataset : datasets/posprivacy/PRISM Abstract Programmatic video generation through code offers geometric precision and temporal coherence beyond pixel-level diffusion models, yet rigorously evaluating whether language models can produce spatially correct animated outputs remains an open problem. We introduce PRISM , a large-scale benchmark of 10,372 human-calibrated instruction-code pairs ( 20 × 20\times larger than prior programmatic video generation benchmarks), grounded in real-world knowledge visualization scenarios across English and Chinese and spanning 437 subject categories. We further propose a funnel-style evaluation framework with four complementary metrics: Code-Level Reliability for executability, Spatial Reasoning for layout correctness over full animation sequences, and Prompt-Aware Dynamic Visual Complexity (PADVC) and Temporal Density (TD) for diagnosing dynamic expression and temporal activity. Systematic evaluation of seven mainstream LLMs reveals a striking Execution-Spatial Gap : the average drop from execution success rate to spatial pass rate is approximately 41%, showing that runnable code does not necessarily yield spatially coherent visual output. These findings show that programmatic video generation evaluation should go beyond executability. PRISM provides a principled benchmark for advancing spatially coherent code generation. † † footnotetext: † Equal contribution. † † footnotetext: 🖂 Corresponding author. 1 Introduction (a) Pixel-level. (b) Programmatic. Figure 1 : Qualitative contrast between pixel-level and programmatic video generation. Recent advances in large language models (LLMs) and generative AI have broadened automated content creation from text and images to videos [ 25 , 61 , 59 ] . Automated video generation has since evolved along two major routes [ 61 ] . Pixel-level methods, typically based on diffusion models, achieve impressive visual fidelity by modeling videos in pixel space [ 22 , 46 ] . Without explicit geometric and symbolic constraints, they still struggle to maintain spatial consistency, accurate text and symbol rendering, and long-horizon logical coherence [ 11 , 52 ] . In contrast, programmatic methods execute scripts to define objects, layouts, transformations, and temporal dynamics [ 17 , 50 ] . Grounded in executable code, this paradigm offers stronger geometric precision, controllability, and temporal consistency [ 10 , 54 ] . These strengths make it particularly suitable for knowledge-intensive scenarios such as educational animation, knowledge visualization, and scientific demonstration, where preserving spatial layouts, temporal dynamics, and symbolic semantics is more critical than merely achieving photorealistic appearance [ 59 , 60 , 23 , 9 ] , as demonstrated by the qualitative contrast between pixel-level and programmatic generation in Figure 1 . Scaling programmatic video generation depends on whether current LLMs can synthesize video programs that are both executable and spatially and temporally correct [ 65 , 38 , 28 , 19 ] . Existing benchmarks remain insufficient for systematically evaluating this capability [ 42 , 51 ] . First, existing visual code generation tasks mostly focus on static outputs, such as charts, webpages, SVGs, or single-frame layouts [ 57 , 64 , 21 ] , whereas programmatic video generation requires models to maintain consistency in object positions, relative relations, and motion trajectories over continuous time [ 50 , 7 ] . Second, existing programmatic video datasets are typically small in scale [ 23 , 43 ] , making it difficult to cover the diverse object compositions, spatial relations, and dynamic processes found in real-world scenarios. More importantly, existing evaluation protocols leave critical blind spots: code executability mainly measures whether a program can run [ 14 , 15 , 53 , 29 ] , while coarse-grained VLM-as-judge evaluation provides only holistic judgments of the rendered video [ 35 , 1 , 31 ] . Neither can reliably capture fine-grained rendering failures that appear only in a few key frames, such as layout misalignment and out-of-bounds objects. Figure 2 : Data overview of PRISM and aggregate model capability on the benchmark . The left panel illustrates multi-level subject coverage with representative examples, while the right panel presents direction-aligned scores summarizing model capability. To address these issues, we introduce PRISM ( P rogrammatic R easoning I n S patial M odalities) (See Figure 2 ), a large-scale bilingual benchmark. PRISM instantiates programmatic video generation with Manim 1 1 1 Manim is a widely used Python package for programmatic video generation [ 44 ] . and aims to characterize and bridge the Execution-Spatial Gap : code may be executable, its rendered output can still contain spatial-layout and dynamic-process errors. Grounded in real scenarios, PRISM uses a hierarchical data curation framework that integrates task collection, code generation, automated filtering, and human revision. From more than 30,000 raw candidates, the pipeline filters out samples with execution failures, invalid spatial layouts, incomplete animations, or instruction-code mismatches, resulting in 10,372 executable, visually coherent, and instruction-aligned instruction-code pairs across English and Chinese subsets. These curated samples support reliable evaluation and provide high-quality data resources for future LLM training in programmatic video generation. Our principal contributions are summarized as follows: • Large-Scale Human-Calibrated Dataset : We release PRISM, comprising 10,372 high-quality instruction-code pairs across English and Chinese subsets. Grounded in real scenarios and curated through a hierarchical quality-control framework, PRISM is 20\times larger than prior programmatic video generation benchmarks [ 23 , 33 ] . • Multi-Dimensional Evaluation : We design an automated evaluation framework with four complementary dimensions: Code-Level Reliability measures execution robustness; Spatial Reasoning assesses layout planning; Prompt-Aware Dynamic Visual Complexity (PADVC) quantifies input-relevant dynamic richness; and Temporal Density (TD) captures visual information density. • Comprehensive LLM Benchmarking : We conduct a large-scale and comprehensive evaluation of seven representative open- and closed-source LLMs across both subsets, complemented by further analyses and ablation studies. The results reveal a severe Execution-Spatial Gap . Further analyses show that excessive dynamic intensity degrades spatial quality, and that extended thinking modes fail to bring reliable spatial gains, suggesting spatial planning as the key bottleneck. 2 PRISM 2.1 Hierarchical Data Curation Framework PRISM is grounded in authentic knowledge visualization scenarios [ 9 , 10 ] . We collect English and Chinese outline fragments spanning diverse knowledge concepts and instantiate them as Manim programs. Manim is well suited to this setting, as it precisely maps Python instructions onto an absolute coordinate system, enabling explicit control over object placement, geometric relations, and temporal animations [ 17 ] . Although PRISM adopts Manim as its instantiation medium, its collection pipeline and subsequent evaluation protocol are decoupled from Manim, making the framework broadly generalizable [ 50 ] . We then develop a collaborative data construction workflow that combines agent-based draft generation, annotator revision, and expert quality prescreening [ 23 , 43 ] . This standardized production paradigm yields a raw pool of more than 30,000 candidate samples [ 8 , 24 ] . Since PRISM targets precise spatial relationships among objects across consecutive frames, including subtle failures such as unexpected occlusion and local out-of-frame placement, it requires a highly reliable, scalable quality-assurance mechanism. [ 16 , 47 , 20 ] . VLMs are ill-suited to this role, given their known weaknesses in fine-grained spatiotemporal reasoning [ 45 , 6 ] , susceptibility to hallucinations [

Read the complete text