VisNumBench: Evaluating Number Sense of Multimodal Large Language Models

Tengjin Weng, Jingyi Wang, Wenhao Jiang, Zhong Ming · 2025

Read VisNumBench: Evaluating Number Sense of Multimodal Large Language Models by Tengjin Weng, Jingyi Wang, Wenhao Jiang, Zhong Ming (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-14939.txt.

Description

Weng, Wang, Jiang, and Ming build in twenty twenty-five a benchmark to test whether vision-language models have number sense, the gut feel for amounts that humans show. Their set holds close to two thousand multiple-choice questions drawn from synthetic and real images. It spans seven kinds of visual number traits and four styles of estimation task.

Seventeen leading models, open and commercial, scored far below human level across the board. Models tuned for maths or step-by-step reasoning did no better on number sense. Bigger, broadly stronger models gained only modestly. The authors release the data and code openly. They hope the benchmark pushes work on a basic skill that current models plainly lack.

Catalog record

CREATORTengjin Weng, Jingyi Wang, Wenhao Jiang, Zhong Ming
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-14939

Browse the full library · About the Tower of Babel