VF-Eval : Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos

Tingyu Song, Tongyan Hu, Guo Gan, Yilun Zhao, School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, National University of Singapore, Zhejiang University, Yale University · 2025

Read VF-Eval : Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos by Tingyu Song, Tongyan Hu, Guo Gan, Yilun Zhao, School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, National University of Singapore, Zhejiang University, Yale University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23693.txt.

Description

Song, Hu, Gan, and Zhao build a benchmark that grades AI models on critiquing machine-made videos. Most video tests use natural footage, yet models are also asked to judge generated clips, a skill barely measured. Their 2025 benchmark, VF-Eval, sets four tasks: checking story coherence, noticing errors, naming the error type, and reasoning about the clip.

Thirteen leading multimodal models took the test, and even the best one faltered across tasks. A side experiment, RePrompt, shows that tuning a model toward human feedback improves the videos it helps create. The authors offer the benchmark as a yardstick for a growing blind spot. They frame critique skill as a lever for better generation.

Catalog record

CREATORTingyu Song, Tongyan Hu, Guo Gan, Yilun Zhao, School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, National University of Singapore, Zhejiang University, Yale University
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23693

Browse the full library · About the Tower of Babel