VisNumBench: Evaluating Number Sense of Multimodal Large Language Models
Tengjin Weng, Jingyi Wang, Wenhao Jiang, Zhong Ming · 2025
Read VisNumBench: Evaluating Number Sense of Multimodal Large Language Models by Tengjin Weng, Jingyi Wang, Wenhao Jiang, Zhong Ming (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14939.txt.
Description
Weng, Wang, Jiang, and Ming build in twenty twenty-five a benchmark to test whether vision-language models have number sense, the gut feel for amounts that humans show. Their set holds close to two thousand multiple-choice questions drawn from synthetic and real images. It spans seven kinds of visual number traits and four styles of estimation task.
Seventeen leading models, open and commercial, scored far below human level across the board. Models tuned for maths or step-by-step reasoning did no better on number sense. Bigger, broadly stronger models gained only modestly. The authors release the data and code openly. They hope the benchmark pushes work on a basic skill that current models plainly lack.
Catalog record
| CREATOR | Tengjin Weng, Jingyi Wang, Wenhao Jiang, Zhong Ming |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-14939