Video Text Preservation with Synthetic Text-Rich Videos
Ziyang Liu, Kevin Valencia, Justin Cui · 2025
Read Video Text Preservation with Synthetic Text-Rich Videos by Ziyang Liu, Kevin Valencia, Justin Cui (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-05573.txt.
Introduction
Ziyang Liu, Kevin Valencia, and Justin Cui tackle a weak spot of text-to-video models: words inside the video often come out garbled. Even short phrases can break apart or drift over time. Past fixes tend to be heavy and costly to run.
Their approach builds training data in a cheap, synthetic way. First a text-to-image model makes pictures rich in text. Then an image-to-video model animates them without seeing the text. Those pairs fine-tune a video model with no change to its design. Short text became clearer and steadier across frames, and longer text showed early structure, hinting that weak synthetic supervision can teach better lettering.
Catalog record
| CREATOR | Ziyang Liu, Kevin Valencia, Justin Cui |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 14 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-05573
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Video Text Preservation with Synthetic Text-Rich Videos Ziyang Liu, Kevin Valencia, Justin Cui Abstract While Text-To-Video (T2V) models have advanced rapidly, they continue to struggle with generating legible and coherent text within videos. In particular, existing models often fail to render correctly even short phrases or words and previous attempts to address this problem are computationally expensive and not suitable for video generation. In this work, we investigate a lightweight approach to improve T2V diffusion models using synthetic supervision. We first generate text-rich images using a text-to-image (T2I) diffusion model, then animate them into short videos using a text-agnostic image-to-video (I2v) model. These synthetic video-prompt pairs are used to fine-tune Wan2.1, a pre-trained T2V model, without any architectural changes. Our results show improvement in short-text legibility and temporal consistency with emerging structural priors for longer text. These findings suggest that curated synthetic data and weak supervision offer a practical path toward improving textual fidelity in T2V generation. 1 Introduction Diffusion models, since its inception, have become the primary method for text-to-image and text-to-video generation. Despite these advancements, however, text-to-video diffusion models still struggle in generating legible text. Generated images and videos often contain unnecessary and impurities in the letters of the rendered text, leading to a decrease in overall quality. Attempts have been made to remedy this issue for text-to-image diffusion models, using various techniques such as using a specific text encoder to encode rendered text for generation [ 11 , 12 ] , large language models to encode positional and textual information [ 2 ] , and fine-tuning specifically on images with rendered text [ 25 ] , among many other techniques. These techniques have varying levels of success in improving quality of rendered text in images. However, there have not been much exploration in techniques to improve text rendering for text-to-video diffusion models. Due to how computationally expensive text-to-video diffusion models are to train and run inference, many of the techniques used to improve the quality of rendered text in text-to-image diffusion models are infeasible for text-to-video models. Our aim is to explore less computationally expensive approaches to improve text-to-video diffusion models in text rendering. Our approach to this problem is by fine-tuning a text-to-video diffusion model over a curated dataset of videos generated by OpenAI’s Sora which contain coherent text without artifacts and their corresponding prompts. Our contribution: From our experiments, we find that it suffices to fine-tune existing text-to-video models on a curated dataset of videos featuring text and prompts describing the video and the text within. By fine-tuning our text-to-video diffusion model, we are able to produce videos which are able to generate text which are coherent and free from artifacts. 2 Related works 2.1 Diffusion Models for Image and Video Generation Diffusion models have been a powerful generative modeling framework, dominating fields like image and video generation. Starting from Denoising Diffusion Probabilistic Models (DDPMs) [ 5 , 6 ] , they have been successfully applied to both image and video generation. In text-to-image (T2I) generation, models like GLIDE, DALL·E 2, and Stable Diffusion [ 15 ] produce high-resolution images aligned with text prompts and can often render legible text when guided carefully. Their ability to preserve high-frequency details makes them a natural starting point for tasks requiring precise spatial structure. Recent extensions of diffusion to text-to-video (T2V) generation include zero-shot pipelines such as Text2Video-Zero and ControlVideo, and end-to-end models like Video Diffusion Models (VDM), Imagen Video, and Make-A-Video [ 9 , 24 , 6 , 15 ] . While these methods achieve realistic motion and temporal consistency, they frequently fail to generate accurate or readable text across frames. In parallel, identity preservation has emerged as a key benchmark for visual consistency in video generation. Methods like CustomVideo and ConsisID [ 20 , 22 ] aim to preserve subject-specific features, often using frequency-aware designs. We build on Wan2.1 [ 3 ] , which achieves state-of-the-art identity preservation through enhanced temporal modeling and attention-based consistency. Although not designed specifically for text, Wan2.1’s ability to retain high-frequency details across frames makes it a strong foundation for improving textual fidelity in generated videos. 2.2 Diffusion for text rendering Recent work has addressed the challenge of preserving readable text in image generation by introducing structure-aware and OCR-guided enhancements. ARTIST [ 23 ] proposes a two-stage framework where a specialized ”text diffusion model” first generates text layouts at the glyph level, which are then used to guide a visual diffusion model to produce high-quality, legible text within realistic scenes. This separation of layout and appearance helps prevent artifacts and improves readability. TextDiffuser [ 1 ] also adopts a staged approach: a transformer-based layout planner predicts character positions, and a diffusion model renders images conditioned on these layouts. It introduces a glyph-level OCR loss and trains on MARIO-10M, a large-scale dataset curated specifically for visual text generation. In contrast, AnyText [ 17 ] focuses on multilingual and editable text rendering by incorporating text control tokens and perceptual losses derived from OCR feedback. It leverages an auxiliary latent space to enhance stroke-level accuracy, resulting in more precise and flexible text generation across diverse scripts and fonts. Together, these methods demonstrate that combining structural planning with OCR-aware supervision is effective in improving the clarity and correctness of generated text in images. While significant progress has been made in preserving readable text in image generation through OCR-guided training, layout planning, and glyph-level losses, these strategies have not yet been widely adopted or explored in the context of video generation. Current text-to-video (T2V) models primarily focus on visual realism and temporal coherence, often at the cost of structured text fidelity. To our knowledge, there is no existing work that explicitly targets the problem of rendering accurate and consistent text across frames in T2V. Our work aims to bridge this gap by adapting a high-performing identity-preserving video model to address the unique challenges of text consistency and legibility in generated videos. 2.3 Fine-tuning diffusion models Due to the computational cost of training text-to-video diffusion models, a common approach to tailor text-to-video diffusion models for certain tasks is to take a pre-trained text-to-video diffusion model’s weights, and given a small, curated dataset of videos and prompts, train the model to reproduce the video given the prompt. Previous works have been done in fine-tuning text-to-image diffusion models for tasks such as better resemblance to sample images (ID Preservation) [ 13 ] , fairness [ 14 ] , improving image quality [ 19 , 21 ] , among other tasks. Fine tuning text-to-image and text-to-video diffusion models allow for a much less computationally expensive alternative to There are various approaches to fine-tuning diffusion models, varying from strictly reproducing the output provided given a prompt, or using rewards to fine-tune the diffusion models, using directly defined rewards [ 4 ] or reward models [ 18 ] or other metrics such as human feedback or self-play [ 19 , 21 ] . Further improvements to computational efficiency can be made by using Low-Rank Adaption (LoRA) [ 7 , 16 , 8 ] . Initially designed for large language models, LoRA allows for more parameter-efficient fine tuning by approximating the changes to weight matrices of diffusion models with the product of two