Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Models
Xuyang Guo, Zekai Huang, Jiayan Huo, Yingyu Liang, Zhenmei Shi, Zhao Song, Jiahao Zhang · 2025
Read Can You Count to Nine? A Human Evaluation Benchmark for Counting Limits in Modern Text-to-Video Models by Xuyang Guo, Zekai Huang, Jiayan Huo, Yingyu Liang, Zhenmei Shi, Zhao Song, Jiahao Zhang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-04051.txt.
Description
Xuyang Guo and co-authors ask today's best video generators a child's question: show me exactly this many. Their benchmark, T2VCountBench, orders clips with set counts of objects and has human judges tally what actually appears on screen. Open and commercial text-to-video systems all take the test, under varied styles, motion, and prompt languages.
Every model stumbled. Counts of nine or fewer, trivial for a person, already broke them; outputs drifted above or below the order with no reliability. Splitting prompts into smaller steps did not rescue the numbers. Style and motion nudged results but never fixed them. The authors frame counting as a basic obedience test that generative video still fails, and a clear target for the next round of models.
Catalog record
| CREATOR | Xuyang Guo, Zekai Huang, Jiayan Huo, Yingyu Liang, Zhenmei Shi, Zhao Song, Jiahao Zhang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-04051