Your VLM is not good at counting pushups

Shengzhi Li, Jiarun Chen, Karun Sharma, Jiaqi Su, Shichao Pei · 2026

Read Your VLM is not good at counting pushups by Shengzhi Li, Jiarun Chen, Karun Sharma, Jiaqi Su, Shichao Pei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23407.txt.

Description

Li and four colleagues introduce PushupBench, 446 long-form clips averaging about 37 seconds for evaluating repetition counting in video. Vision-language models recognise what happens but fail to count how many times: the best frontier model reaches 42.1 percent exact accuracy, while open 4B models score about 6 percent, matching supervised baselines. Accuracy alone misleads, since weaker models lean on the modal count instead of reasoning over time. Fine-tuning on counting with 1,000 samples transfers to general video understanding, lifting three benchmarks, suggesting counting proxies broader temporal reasoning.

Catalog record

CREATORShengzhi Li, Jiarun Chen, Karun Sharma, Jiaqi Su, Shichao Pei
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35777
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.23407

Browse the full library · About the Tower of Babel