Evaluating Design Video Generation: Metrics for Compositional Fidelity
Adrienne Deganutti, Dingning Cao, Jaejung Seol, Elad Hirsch, Purvanshi Mehta · 2026
Read Evaluating Design Video Generation: Metrics for Compositional Fidelity by Adrienne Deganutti, Dingning Cao, Jaejung Seol, Elad Hirsch, Purvanshi Mehta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16223.txt.
Introduction
Deganutti and four colleagues provide a fully automated test framework for design animation video generation, a field with no standard benchmark. Design animation sets strict rules: set parts must move with set motion, direction, speed, and timing, while other regions stay stable and layout holds. Their framework spans four facets: layout fidelity, motion correctness, temporal quality, and content fidelity. It removes reliance on subjective human judgment and sets a common basis for tracking progress.
Catalog record
| CREATOR | Adrienne Deganutti, Dingning Cao, Jaejung Seol, Elad Hirsch, Purvanshi Mehta |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68130 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16223
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Design Video Generation: Metrics for Compositional Fidelity Adrienne Deganutti, Dingning Cao, Jaejung Seol, Elad Hirsch, Purvanshi Mehta Abstract Generative video models are increasingly used in design animation tasks, yet no standardized evaluation framework exists for this domain. Unlike natural video generation, design animation imposes structured constraints: specific components shall animate with prescribed motion types, directions, speed and timing, while non-animated regions must remain stable and layout structure must be preserved. This paper provides a fully automated evaluation framework organized across four dimensions: layout fidelity, motion correctness, temporal quality, and content fidelity. This eliminates the reliance on subjective human evaluation and establishes a common basis for benchmarking progress in the field. We release the code and dataset here: https://github.com/purvanshi/lica-bench . 1 Introduction Design animations are a cornerstone of modern digital communication. Producing them requires specialized expertise: a designer must select which components to animate, choose appropriate motion types (e.g., fade, slide, scale, rotate), specify direction and timing, and ensure that the overall composition remains coherent throughout. Recent advances in generative video models have opened the door to automating this workflow, with models such as Sora ( 15 ) and Veo ( 6 ) now being applied to design animation tasks where a static layout and per-component motion specification are provided as input. Yet early benchmarking results reveal systematic shortcomings: models struggle to ground animation instructions to the correct visual regions, frequently applying a single dominant motion globally, animating the wrong elements, or hallucinating motion unrelated to the prompt. These failures underscore that design animation is not a subproblem of natural video synthesis but a distinct task with structured, constraint-driven requirements — specific components must animate with prescribed motion types and timing while non-animated regions remain stable and the spatial layout is preserved. Figure 1 : Overview of the proposed framework. We evaluate design videos across four dimensions: motion type, motion direction, duration, and text recoverability. Despite this progress, evaluation of generated design animations remains ad hoc: most work reports Fréchet Video Distance or relies on small-scale human studies, neither of which captures the structured requirements unique to design animation, namely that specific components must move in prescribed ways while the rest of the composition remains intact. In this paper, we propose a fully automated evaluation framework organized along four complementary dimensions: layout fidelity, motion correctness, temporal quality, and content fidelity, that together provide a comprehensive and reproducible measure of design animation quality, eliminating the need for costly subjective assessment and establishing a common benchmark for the field. Our key contributions are as follows. • The first dimension-decomposed evaluation framework for design animation, where each axis is independently grounded in the layout specification. • Within motion correctness, a suite of four sub-metrics: motion type, direction, animation duration, and component visible duration, with ground-truth signals automatically extracted from the layout metadata. • A two-track benchmark of 86 single-component clips (isolating one element on a uniform background) and 136 full-layout templates ( 894 component evaluations, {\sim}6.6 per template), spanning 15 animation types across image, text, and group component families. • Empirical validation showing that our framework recovers ground-truth motion attributes with 87.8\% accuracy on single-component renders using a transparent, rule-based pipeline with no learned classifier, and a comparative study of Sora-2 and Veo-3.1 revealing that the two models exhibit sharply different failure modes along axes that aggregate perceptual scores cannot separate. 2 Related Works Graphic Design Datasets and Benchmarks. Static graphic design research has produced a range of layout datasets, from mobile UI collections such as Rico ( 4 ) to multi-layer template datasets like Crello ( 16 ) , poster benchmarks such as PosterLayout ( 8 ) and CGL ( 20 ) , and e-commerce banner datasets ( 17 ) . On the evaluation side, DesignProbe ( 11 ) benchmarks MLLMs on color, typography, and layout understanding, while AesEval-Bench ( 1 ) targets aesthetic quality assessment and Graphic-Design-Bench ( 3 ) grounds tasks in full structural metadata to enable layer-aware evaluation. Recent visual-text benchmarks further show that text rendering remains a distinct failure mode for generative models: OCRGenBench ( 18 ) evaluates OCR-related generative capabilities across document, scene-text, artistic-text, and layout-rich settings, and PosterCraft ( 2 ) emphasizes that high-quality poster generation requires both accurate text rendering and aesthetic layout composition. Except for ( 3 ) , these benchmarks operate on static images, leaving the temporal and motion dimensions of graphic design unaddressed. Video Generation and Understanding Benchmarks. General video benchmarks fall into two tracks. VBench ( 9 ) decomposes generation quality into 16 hierarchical dimensions, including subject consistency, motion smoothness, and temporal flickering. validated against human preference annotations. EvalCrafter ( 13 ) evaluates text-to-video models across visual, content, and motion quality dimensions using 700 prompts derived from real-world user data. For multimodal understanding, Video-MME ( 5 ) spans six primary visual domains with 30 subfields and varying video durations to stress-test temporal reasoning in MLLMs. These benchmarks target naturalistic or cinematic content and do not account for the compositional, typographic, or brand-driven properties of designed video. Advertisement and Creative Video. A growing set of works recognizes the distinct challenges of commercial video content. VideoAds ( 19 ) benchmarks MLLMs on advertisement videos with complex temporal structures across visual finding, summarization, and reasoning tasks, finding that even the strongest models fall well short of human performance. AdsQA ( 14 ) provides a QA benchmark derived from over 1,500 ad videos across five tasks centered on the persuasive structure of advertising. E-VAds ( 12 ) targets e-commerce short videos, identifying a gap in benchmarks that address commercial intent reasoning rather than general action recognition or commonsense QA. While these works probe semantic and narrative understanding of ads, none evaluate the design-specific dimensions: layout, typography and motion composition, that our benchmark is built around. 3 Dataset Evaluating design video generation at the component level requires ground-truth data that existing video benchmarks do not provide: per-component bounding boxes, animation type and timing metadata, and deterministic reference renders against which metrics can be validated. Natural-video datasets lack layered structure entirely, and even GDB’s animation evaluation set ( 3 ) comprises only 10 samples per generation task with human-only evaluation, making it insufficient for automated metric development. We therefore construct a dedicated evaluation dataset from the LICA layered-composition corpus ( 7 ) , which uniquely supplies the full component-level metadata our framework requires. Specifically, we draw from the public release lica-dataset and filter to entries whose sub-category is videos , yielding 136 source layouts spanning 66 unique templates. From each layout we extract the canvas specification (dimensions, background color, total duration), and for every component its type (text, image, or group), bounding box, and animation attributes including motion type (drawn from LICA’s 31-type taxonomy), direction, animation duration, trigger time, and visible duration. Each animated component carries exactly one motion type assigned at authoring time in LICA; we extract these assignments verbatim and do not synthesize or reassign animations. For text components we additionally extract the rendered string, font family, font size, weight, style, color, line height, letter spacing, and alignment. A component is treated as animated when it carries a non-empty animations list; component trees are walked recursively so that animated elements nested inside groups are not