Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench

BAAI FlagEval Team · 2025

Read Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench by BAAI FlagEval Team (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-26865.txt.

Introduction

Fenfen Lin, with nine co-authors, benchmarks how well vision-language models read measuring instruments. People read gauges and rulers with little training, but the team finds current models struggle. Their benchmark, MeasureBench, covers real and synthesised images of many measurement types.

Lin and the team also build a pipeline that procedurally generates gauges with controlled pointers, scales, fonts, lighting, and clutter, so data can scale. Even the strongest frontier models did poorly at measurement reading in general. A round of reinforcement finetuning on synthetic data improved results significantly, both on synthetic and real images. The authors read the failure as a limit in fine-grained spatial grounding, and they release the resource to help close the gap between recognising numbers and measuring the world.

Catalog record

CREATORBAAI FlagEval Team
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-26865

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench BAAI FlagEval Team Abstract Reading measurement instruments is effortless for humans and requires relatively little domain expertise, yet it remains surprisingly challenging for current vision-language models (VLMs) as we find in preliminary evaluation. In this work, we introduce MeasureBench, a benchmark on visual measurement reading covering both real-world and synthesized images of various types of measurements, along with an extensible pipeline for data synthesis. Our pipeline procedurally generates a specified type of gauge with controllable visual appearance, enabling scalable variation in key details such as pointers, scales, fonts, lighting, and clutter. Evaluation on popular proprietary and open-weight VLMs shows that even the strongest frontier VLMs struggle measurement reading in general. A consistent failure mode is indicator localization: models can read digits or labels but misidentify the key positions of pointers or alignments, leading to big numeric errors despite plausible textual reasoning. We have also conducted preliminary experiments with reinforcement learning over synthetic data, and find encouraging results on in-domain synthetic subset but less promising for real-world images. Our analysis highlights a fundamental limitation of current VLMs in fine-grained spatial grounding. We hope this resource can help future advances on visually grounded numeracy and precise spatial perception of VLMs, bridging the gap between recognizing numbers and measuring the world. TL;DR: Fine-grained visual understanding tasks such as visual measurement reading have been surprisingly challenging for frontier general-purpose vision-language models. We introduce MeasureBench, a benchmark with diverse images of measuring instruments collected from both real-world images and a new data synthesis pipeline. 1 Introduction Recent advances in vision-language models (VLMs) have demonstrated impressive capabilities in tackling complex reasoning tasks that combine textual and visual information. Models or systems such as GPT-5 ( OpenAI 2025a ) and Gemini 2.5 Pro ( Gemini Team 2025 ) achieve human-expert level performance on college-level problems in MMMU ( Yue et al. 2024 ) and MMMU-Pro ( Yue et al. 2025 ) . Even on Humanity’s Last Exam ( Phan et al. 2025 , HLE; ) , a benchmark at the frontier of human knowledge, state-of-the-art models achieve accuracies exceeding 25%, substantially surpassing the human average. Beyond academic-style evaluations, VLMs have also been applied to real-world scenarios such as embodied intelligence and autonomous driving, where success relies more heavily on precise visual perception than on complex logical reasoning. That said, state-of-the-art vision–language models (VLMs) still struggle with fine-grained perception—e.g., low-level visual cues, precise geometry, and subtle changes—even when their high-level reasoning appears strong. Existing fine-grained evaluations are well represented by text reading and chart reasoning ( Singh et al. 2019 ; Masry et al. 2022 ; Tang et al. 2025 ) , or by similarly artificial low-level vision tests such as BlindTest ( Rahmanzadehgervi et al. 2024 ) and SalBench ( Dahou et al. 2025 ) . However, they rarely require mapping physical scales to numeric values. Visual instrument reading tasks usually require fine-grained visual perception, light quantitative reasoning, and basic arithmetic. Examples include reading pressure gauges in industrial settings, and thermometers or even as simple as clocks in daily life. Accurate interpretation of these instruments is crucial for safety, efficiency, and decision-making across domains for vision language models or future embodied AI systems. While a few exist studies have covered very specific types of reading such as clocks ( Saxena et al. 2025 ; Yang et al. 2022 ) , rulers ( Matuzevičius 2023 ; Pan et al. 2025 ) , industrial gauges ( Izquierdo-Domenech et al. 2025 ; Valente et al. 2025 ) , and household meters ( Van et al. 2025 ) , they do not span the diversity of instruments or reading designs. Figure 1: Sampled MeasureBench examples real-world set, these four reading designs are commomly used in various measuring instruments. To fill this gap, we introduce MeasureBench , a benchmark for evaluating VLMs on measuring instrument reading across 26 instrument types and four types of readout designs. Each image is paired with a reading question. MeasureBench comprises 2,442 image–question pairs: 1,272 diverse real-world images collected and human-annotated, and 1,170 synthetic images generated with randomized readings for 39 instruments. Our data synthesis pipeline has two complementary backends: (i) a 2D programmatic renderer for diverse layouts with full control over fonts and geometry; and (ii) a 3D Blender renderer for photorealistic scenes with realistic lighting, materials, reflections, and occlusions. The pipeline is fully automated and readily scalable in both breadth (instrument types) and depth (variations). This pipeline can be used to generate additional data for training or evaluation. We evaluate a number of modern VLMs on MeasureBench and report these key findings: • Persisting difficulty. Current VLMs still struggle with instrument reading, with the best model achieving only 30.3% accuracy on the real-world set and 26.1% on the synthetic set. • Object recognition and text reading seems easy, but inferring numbers is hard. Models exhibit strong image understanding and text recognition—e.g., reading units—reaching over 90% accuracy on unit identification. Yet they falter on mapping scales to numeric values. • Systematic fine-grained errors. Models often “know how to read” but miss details: They misinterpret pointer positions, confuse adjacent ticks, and mismatch values to scale markings, leading to near-miss but incorrect answers. With our data synthesis pipeline that produces accurately annotated readings, we have also conducted preliminary experiments of reinforcement learning using synthetic data. Results are encouraging in that the synthetic subset of MeasureBench can get significantly improved, but not as promising on real-world images. In summary, this work makes the following contributions: • We present MeasureBench , a comprehensive benchmark targeting fine-grained instrument reading across 26 instrument types and 2,442 image–question pairs. • We provide a controllable 2D/3D synthesis pipeline that produces precise labels for sketch or photorealistic images with randomized readings for 39 instruments. • We deliver a standardized evaluation of 17 contemporary VLMs and an analysis of their failure modes, highlighting concrete gaps in low-level perception and precise geometric reasoning that are not bridged by language priors. • Based on our synthetic data pipeline, we conduct preliminary experiments of reinforcement learning, leading to some promising results for more data curation but also implying a potential need for future efforts on more crafted design of visual representation for better generalization. Together, these results position MeasureBench as a focused test for the fine-grained visual competencies. We hope our results and this new benchmark could help future modeling efforts towards more fine-grained visual representation that would enable future VLMs to reason from detailed visual clues, then naturally generalize on instrument reading and other visual tasks that requires crucial visual capabilities such as geometric alignment and spatial understanding. 2 MeasureBench 2.1 Overview of MeasureBench We introduce MeasureBench , a comprehensive benchmark for evaluating the ability to read values from measuring instruments. MeasureBench comprises two main components: (i) a diverse dataset of instrument images with standardized annotations, and (ii) a data synthesis framework for generating additional training and evaluation data. By visual appearance, we categorize instruments into four readout designs (see also Figure 1 for examples from the real-world images in MeasureBench): • Dial : Analog instruments with one or more pointers (e.g., ammeters and pressure gauges which typically have a single pointer, whereas clocks often have two

Read the complete text