VF-Eval : Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos
Tingyu Song, Tongyan Hu, Guo Gan, Yilun Zhao, School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, National University of Singapore, Zhejiang University, Yale University · 2025
Read VF-Eval : Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos by Tingyu Song, Tongyan Hu, Guo Gan, Yilun Zhao, School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, National University of Singapore, Zhejiang University, Yale University (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23693.txt.
Description
Song, Hu, Gan, and Zhao build a benchmark that grades AI models on critiquing machine-made videos. Most video tests use natural footage, yet models are also asked to judge generated clips, a skill barely measured. Their 2025 benchmark, VF-Eval, sets four tasks: checking story coherence, noticing errors, naming the error type, and reasoning about the clip.
Thirteen leading multimodal models took the test, and even the best one faltered across tasks. A side experiment, RePrompt, shows that tuning a model toward human feedback improves the videos it helps create. The authors offer the benchmark as a yardstick for a growing blind spot. They frame critique skill as a lever for better generation.
Catalog record
| CREATOR | Tingyu Song, Tongyan Hu, Guo Gan, Yilun Zhao, School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences, National University of Singapore, Zhejiang University, Yale University |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23693