T2VTextBench: A Human Evaluation Benchmark for Textual Control in Video Generation Models

Xuyang Guo and Jiayan Huo and Zhenmei Shi and Zhao Song and Jiahao Zhang and Jiale Zhao · 2025

Read T2VTextBench: A Human Evaluation Benchmark for Textual Control in Video Generation Models by Xuyang Guo and Jiayan Huo and Zhenmei Shi and Zhao Song and Jiahao Zhang and Jiale Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-04946.txt.

Introduction

Video generators can conjure sweeping scenes from a sentence, yet Xuyang Guo and five co-authors ask a narrower question: can they write? Their benchmark, T2VTextBench, is the first human-judged test of on-screen text in generated video, from captions to math formulas. Its prompts pair complex text strings with changing scenes, so a model must keep lettering exact and steady across frames.

The team ran ten leading systems, open and commercial, through the suite and had people grade the results. Most systems failed, producing text that was garbled, unstable, or illegible. The authors call precise text a blind spot of current video generation and offer the benchmark as a yardstick for research aimed at fixing it.

Catalog record

CREATORXuyang Guo and Jiayan Huo and Zhenmei Shi and Zhao Song and Jiahao Zhang and Jiale Zhao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-04946

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

T2VTextBench: A Human Evaluation Benchmark for Textual Control in Video Generation Models Xuyang Guo and Jiayan Huo and Zhenmei Shi and Zhao Song and Jiahao Zhang and Jiale Zhao Abstract Thanks to recent advancements in scalable deep architectures and large-scale pretraining, text-to-video generation has achieved unprecedented capabilities in producing high-fidelity, instruction-following content across a wide range of styles, enabling applications in advertising, entertainment, and education. However, these models’ ability to render precise on-screen text, such as captions or mathematical formulas, remains largely untested, posing significant challenges for applications requiring exact textual accuracy. In this work, we introduce T2VTextBench, the first human-evaluation benchmark dedicated to evaluating on-screen text fidelity and temporal consistency in text-to-video models. Our suite of prompts integrates complex text strings with dynamic scene changes, testing each model’s ability to maintain detailed instructions across frames. We evaluate ten state-of-the-art systems, ranging from open-source solutions to commercial offerings, and find that most struggle to generate legible, consistent text. These results highlight a critical gap in current video generators and provide a clear direction for future research aimed at enhancing textual manipulation in video synthesis. 1 Introduction Text-to-video generative AI [ 52 , 22 , 56 , 62 ] has been a game-changer in real-world applications such as advertising, entertainment, and education purposes. Thanks to recent advances in scalable deep architectures, particularly diffusion models [ 26 , 15 ] and Transformers [ 1 , 42 ] , and large-scale training on text-video pairs harvested from the web, such systems currently exhibit unprecedented instruction-following ability and aesthetic quality across a wide variety of styles. State-of-the-art models such as Sora [ 46 ] and SD Video [ 3 ] are rapidly being adopted as central elements of users’ online experiences, gaining widespread attention and impact. Despite these successes, there is growing concern over text-to-video models’ ability to generate accurate on-screen text [ 47 , 38 ] . In many applications, this limitation can have serious consequences. For instance, in advertising, a brand name must be rendered precisely, and in educational videos, explanatory text or mathematical formulas also should appear correctly. While recent research has significantly improved textual manipulation in text-to-image models [ 11 , 54 , 65 , 49 ] , the corresponding capability in text-to-video remains largely untested. A systematic evaluation of on-screen text fidelity in video generation is therefore essential both for guiding downstream applications and for uncovering fundamental model limitations. To address this gap, we propose T2VTextBench , a comprehensive benchmark for evaluating textual manipulation in modern text-to-video models. Our benchmark stresses both complex textual content and temporal coherence (consistent rendering across frames), probing each model’s ability to follow intricate human instructions. The benchmark also incorporates comprehensive human evaluation, aligning with human preferences and addressing nuances in temporal dynamics. We systematically evaluate ten leading models, including both open-source and proprietary systems, to cover the latest advances in text-to-video generation. Our contributions are summarized as follows: • We propose T2VTextBench, a comprehensive human evaluation on how up-to-date text‐to‐video generators manipulate on‐screen text in videos. • We analyze the impact of temporal text transformations, geometric, visual, and structural, and demonstrate that current text‐to‐video models are unstable with these changes. • We examine the effect of text granularity, finding that models perform well on single words but show significant gaps when generating longer sentences and random individual characters. • We present a cost analysis evaluating the cost‐effectiveness of current models for generating on‐screen text. Roadmap. We systematically review the relevant works of this benchmark in Section 2 . We describe the details of our proposed T2VTextBench benchmark in Section 3 . We show the key assessment results of our benchmark in Section 4 . We present some concluding remarks for this paper in Section 5 . 2 Related Works Visual Text Generation. Generative visual models have achieved unprecedented success in various real-world applications, producing human-preference-aligned images and videos with high fidelity and aesthetic standards [ 24 , 51 , 53 , 35 , 22 , 56 , 62 ] . Despite these advancements, generating accurate and coherent text in visual outputs has increasingly become a significant challenge. One prominent line of research focuses on text rendering in general text-to-image models [ 11 , 61 , 65 , 63 , 66 ] . For instance, TextDiffuser [ 11 ] determines text layout using a Transformer model and generates textual images with diffusion models conditioned on both textual prompts and text layouts, while GlyphControl [ 61 ] enhances text rendering capabilities. Next, SceneVTG [ 65 ] leverages the strong reasoning ability of multimodal LLMs to suggest desirable text layouts and content at various scales and levels for diffusion model conditioning. Another important line of research focuses on typography design and generation using generative AI. These models generate specific textual icons or word art using specially designed architectures, with many early works emphasizing the generation of static images [ 21 , 60 , 47 , 8 ] . Recently, KineTy [ 47 ] introduced a method for generating textual videos with diverse visual effects, employing zero-convolution guidance to control text visibility and glyph loss to ensure readability. Despite previous successes in text manipulation within general text-to-image models and typography-focused models, these approaches either overlook the temporal dynamics of multi-frame videos or fail to generate general-purpose videos beyond typography. Consequently, there remains a substantial gap in text manipulation within text-to-video models, making it highly desirable to benchmark current progress and highlight future research directions. Several pioneering studies have explored text generation in text-to-video models. For example, Wan2.1 [ 2 ] enhances text-to-video models’ intrinsic text generation capabilities through data curation and large-scale training, incorporating hundreds of millions of synthetic text images along with OCR-annotated real image-text pairs. Meanwhile, Text-Animator [ 38 ] proposes a plug-and-play approach to improve the 3D-Unet in existing text-to-video generation models, injecting text embeddings via ControlNet and incorporating a camera control module to embed perspective-related pose information. However, while these models excel at generating relatively short text in videos, they may not adequately address complex temporal dynamics involving motion or text organization. In contrast, our benchmark considers these factors and provides an in-depth evaluation. Benchmarking Text-to-Video Generative Models. Due to the widespread real-world impact of text-to-video models, evaluating their capabilities has become a fundamental research topic. Prior work on benchmarking these generative models has addressed many important aspects, including but not limited to the composition of different properties [ 16 , 50 ] , video fidelity [ 39 ] , temporal coherence [ 41 , 32 ] , object counting [ 18 , 9 ] , physical constraint adherence [ 45 , 19 ] , and storytelling [ 5 ] . Specifically, [ 39 ] proposed a fine-grained evaluation benchmark for text-to-video models, considering three fundamental factors: major content, controllable attributes, and prompt complexity, with manual evaluation conducted on four mainstream text-to-video models. EvalCrafter [ 36 ] introduces a comprehensive benchmark featuring an exhaustive set of prompts, 17 automated objectives, and coverage spanning both image-to-video and text-to-video models. VBench [ 23 , 31 ] evaluates video quality and prompt consistency across 16 human-aligned dimensions, each with tailored prompts and metrics validated against human preferences. Despite the effectiveness of previous benchmarks, most focus on

Read the complete text