FiVE : A Fi ne-grained V ideo E diting Benchmark for Evaluating Emerging Diffusion and Rectified Flow Models
Minghan Li, Chenxi Xie, Yichen Wu, Lei Zhang, Mengyu Wang · 2025
Read FiVE : A Fi ne-grained V ideo E diting Benchmark for Evaluating Emerging Diffusion and Rectified Flow Models by Minghan Li, Chenxi Xie, Yichen Wu, Lei Zhang, Mengyu Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-13684.txt.
Introduction
Editing one object inside a video, swapping its color or shape while the rest flows on untouched, is where video generators are now racing. Minghan Li, Chenxi Xie, Yichen Wu, Lei Zhang, and Mengyu Wang built a benchmark in 2025 to judge that fine-grained skill fairly. It packs one hundred real and generated clips, six edit types, and 420 object-level edit prompts with masks.
Seven editing systems, five diffusion-based and two built on newer flow methods, faced fifteen scoring measures plus a new vision-language check for whether the edit truly landed. The flow-based systems clearly beat the diffusion ones, with the team's own adapted editor on top and least touchy about settings. The authors release the benchmark for the field.
Catalog record
| CREATOR | Minghan Li, Chenxi Xie, Yichen Wu, Lei Zhang, Mengyu Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-13684
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FiVE : A Fi ne-grained V ideo E diting Benchmark for Evaluating Emerging Diffusion and Rectified Flow Models Minghan Li, Chenxi Xie, Yichen Wu, Lei Zhang, Mengyu Wang Abstract Numerous text-to-video (T2V) editing methods have emerged recently, but the lack of a standardized benchmark for fair evaluation has led to inconsistent claims and an inability to assess model sensitivity to hyperparameters. Fine-grained video editing is crucial for enabling precise, object-level modifications while maintaining context and temporal consistency. To address this, we introduce FiVE , a Fi ne-grained V ideo E diting Benchmark for evaluating emerging diffusion and rectified flow models. Our benchmark includes 74 real-world videos and 26 generated videos, featuring 6 fine-grained editing types, 420 object-level editing prompt pairs, and their corresponding masks. Additionally, we adapt the latest rectified flow (RF) T2V generation models—Pyramid-Flow [ 15 ] and Wan2.1 [ 43 ] —by introducing FlowEdit [ 19 ] , resulting in training-free and inversion-free video editing models Pyramid-Edit and Wan-Edit . We evaluate five diffusion-based and two RF-based editing methods on our FiVE benchmark using 15 metrics, covering background preservation, text-video similarity, temporal consistency, video quality, and runtime. To further enhance object-level evaluation, we introduce FiVE-Acc , a novel metric leveraging Vision-Language Models (VLMs) to assess the success of fine-grained video editing. Experimental results demonstrate that RF-based editing significantly outperforms diffusion-based methods, with Wan-Edit achieving the best overall performance and exhibiting the least sensitivity to hyperparameters. More video demo available on the anonymous website: https://sites.google.com/view/five-benchmark 1 Introduction Recent advancements in text-to-image (T2I) and text-to-video (T2V) generation [ 32 , 37 , 1 , 27 , 15 , 25 , 43 ] have led to the emergence of numerous image and video editing methods [ 49 , 10 , 4 , 9 ] , enabling users to create and modify video content with unprecedented flexibility. These methods leverage powerful generative models, such as diffusion models [ 11 , 36 , 6 , 12 ] and Rectified Flow (RF) [ 24 , 8 ] models, to synthesize and edit videos based on textual prompts. However, despite significant progress in these fields, the lack of a standardized benchmark for fair and comprehensive evaluation has still become a major bottleneck hindering further development. Currently, the video editing field has only one publicly available dataset, TGVE [ 50 ] /TGVE+ [ 35 ] ( see Fig . ( 1 )), which includes 76 videos and 6 editing types, such as style transfer, background replacement, single-object and multi-object transformations, as well as object addition and removal. While TGVE+ aims to cover as many editing types as possible, its evaluation within each editing type is not comprehensive, particularly in supporting fine-grained video editing. Fine-grained video editing requires precise object-level modifications while preserving the overall context and temporal consistency of the video, posing higher demands on model performance. Due to the lack of a standardized benchmark and unified evaluation framework for fine-grained video editing, existing methods [ 22 , 29 , 9 , 53 ] often rely on self-collected small-scale video datasets to validate their effectiveness. This approach not only makes it difficult to objectively compare the performance of different methods but also fails to comprehensively assess model robustness to hyperparameters or determine their suitability for real-world applications. Therefore, building a comprehensive and challenging benchmark for fine-grained video editing has become an urgent need to advance the field. To address these challenges, we introduce FiVE ( Fi ne-grained V ideo E diting Benchmark), a comprehensive benchmark designed to evaluate emerging diffusion and rectified flow models for fine-grained video editing. FiVE includes 74 real-world videos and 26 generated videos, covering a diverse range of scenes and editing scenarios. It features 6 fine-grained editing types, 420 object-level editing prompt pairs, and their corresponding masks, providing a rich and challenging testbed for evaluating video editing methods. Additionally, we propose FiVE-Acc, a novel evaluation metric that leverages Vision-Language Models (VLMs) to assess the successful accuracy of fine-grained object-level editing. Specifically, FiVE-Acc complements traditional metrics by providing a more nuanced understanding of editing quality, particularly in terms of semantic alignment, contextual preservation and motion awareness. Recently, many image editing methods [ 33 , 44 , 5 , 19 ] based on the latest rectified flow T2I generation model Flux [ 20 ] have achieved state-of-the-art editing results and fidelity, such as RF-Inversion [ 33 ] , RF-Solver [ 44 ] , and FlowEdit [ 19 ] . However, to the best of our knowledge, there are no video editing methods based on RF T2V generation models, primarily due to the relatively slow development of T2V generation models. Fortunately, the recent release of two state-of-the-art RF-based T2V generation models—Pyramid-Flow [ 15 ] and Wan2.1 [ 43 ] —has opened new possibilities for RF video editing. Building on the latest advancements, we adapt Pyramid-Flow [ 15 ] and Wan2.1 [ 43 ] by introducing FlowEdit [ 19 ] . This adaptation facilitates training-free and inversion-free video editing models, Pyramid-Edit and Wan-Edit, which leverage the strong temporal consistency inherent in video generation models without requiring additional attention mechanisms. Specifically, Pyramid-Flow employs a multi-resolution and temporally autoregressive model architecture across time steps. To adapt to this unique design, we modified FlowEdit to support multi-resolution processing, enabling seamless integration with the Pyramid-Flow framework, leading to Pyramid-Edit. On the other hand, Wan2.1 utilizes the DiT architecture, which unfolds temporal frames into sequences and processes them via self-attention. Since its design is fundamentally similar to image generation models, Wan2.1 can directly adopt FlowEdit without any additional modifications, giving rise to Wan-Edit. To show the effectiveness of the proposed two RF-based editing methods, we compare them with five diffusion-based editing methods on the built FiVE benchmark, evaluating them across 14 metrics, including background preservation, edited text-video similarity, temporal consistency, and generated video quality. Our experiments demonstrate that RF-based editing methods significantly outperform diffusion-based methods across multiple metrics. Notably, Wan-Edit achieves the best overall performance, exhibiting superior editing quality and the least sensitivity to hyperparameters. These findings highlight the potential of RF-based models for fine-grained video editing and provide valuable insights for future research in this area. In summary, this work makes four key contributions: • We introduce FiVE , a comprehensive benchmark for fine-grained video editing, featuring diverse videos, editing types, and evaluation metrics. • We propose FiVE-Acc , a novel metric leveraging VLMs to assess the success of fine-grained editing. • We adapt RF-based T2V models Pyramid-Flow and Wan2.1 using FlowEdit, resulting in efficient and effective video editing methods Pyramid-Edit and Wan-Edit . • We evaluate seven video editing methods across 15 metrics, showing that RF-based editing consistently outperforms diffusion-based approaches. Table 1 : Statistics of existing video editing datasets and benchmarks. Dataset Usage Num. Num. Frames Num. Gen. Src. obj. Edit obj. Obj. Src. obj. Type of Edited Prompts Images Videos Per Video Prompts Videos Words Words Instruct. Masks VIVID-10M Train 672K 73.7K 30 10M ✗ ✗ ✗ ✓ ✓ object, add, remove Señorita-2M Train 0 388.9K 33 \sim 64 2M ✗ ✗ ✗ ✓ ✗ style, object, multiple, color, add, remove, motion TGVE Eval 0 76 32 304 ✗ ✗ ✗ ✗ ✗ style, object, bg, multiple TGVE+ Eval 0 76 32 1417 ✗ ✗ ✗ ✗