VAQUUM: Are Vague Quantifiers Grounded in Visual Data?
Hugh Mee Wong, Rick Nouwen, Albert Gatt · 2025
Read VAQUUM: Are Vague Quantifiers Grounded in Visual Data? by Hugh Mee Wong, Rick Nouwen, Albert Gatt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11874.txt.
Introduction
Hugh Wong, Rick Nouwen, and Albert Gatt check whether vision-language models use vague quantity words the way people do. Their 2025 paper builds VAQUUM, a dataset of over twenty thousand human ratings on quantified statements across more than a thousand images. Words like a few and many bend with context, including how many objects are present.
Models, like people, shift their quantifier use with object counts. Yet across three test styles the models wobble: judging a phrase and producing one seem to run on different machinery. Inconsistency across models adds to the gap. The authors release the dataset for further work. Their findings warn that surface fluency with quantity words hides shaky grounding in what the eyes actually see.
Catalog record
| CREATOR | Hugh Mee Wong, Rick Nouwen, Albert Gatt |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-11874
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
VAQUUM: Are Vague Quantifiers Grounded in Visual Data? Hugh Mee Wong, Rick Nouwen, Albert Gatt Abstract Vague quantifiers such as a few and many are influenced by various contextual factors, including the number of objects present in a given context. In this work, we evaluate the extent to which vision-and-language models (VLMs) are compatible with humans when producing or judging the appropriateness of vague quantifiers in visual contexts. We release a novel dataset, VAQUUM, containing 20,300 human ratings on quantified statements across a total of 1089 images. Using this dataset, we compare human judgments and VLM predictions using three different evaluation methods. Our findings show that VLMs, like humans, are influenced by object counts in vague quantifier use. However, we find significant inconsistencies across models in different evaluation settings, suggesting that judging and producing vague quantifiers rely on two different processes. We release our dataset and code at https://github.com/hughmee/vaquum . 1 Introduction Everyday conversations are replete with statements containing vague quantifiers, such as “There are many horses” (Figure 1 ). Despite the fact that they are vague, they cause surprisingly little misunderstanding among interlocutors ( Jucker et al., 2003 ) . Vague quantifiers, unlike crisp quantifiers, allow for borderline cases in which it is unclear whether the quantifier applies or not, and where we can also expect some variation in the extent to which speakers would use it. For example, all does not allow for borderline cases, but it is unclear when a quantity ceases to be a few or how many many is. Although vague quantifiers have long been a subject of investigation among formal semanticists ( Nouwen, 2010 , see e.g.) and (psycho)linguists ( Moxey and Sanford, 1993a ; van Deemter, 2010 , e.g.) , they have received relatively little attention within the field of natural language processing (NLP). Figure 1: Experiments in this work. We ( 1 ) ask human participants to rate, using a slider, the appropriateness of statements containing vague quantifiers in relation to images. We ( 2 ) extract VLM generation probabilities for those same statements, ( 3 ) prompt the models to generate an accuracy score for them, and ( 4 ) evaluate probabilities assigned to these statements in a multiple-choice setup. The image above is originally from the FSC-147 dataset ( Ranjan et al., 2021 ) . In visually grounded settings, the use of vague quantifiers can be influenced by factors related to the scene itself, such as the number of entities observed ( Coventry et al., 2005 , e.g.) ; their sizes ( Hörmann, 1983 ; Coventry et al., 2010 ) ; as well as information like the speaker’s and hearer’s personal beliefs and attitudes ( Moxey and Sanford, 2000 ; Jucker et al., 2003 ) . This broad range of factors, coupled with their vagueness, raises the question of how well multimodal language models can capture human patterns in the grounded comprehension and use of such expressions. In this paper, we explore this question with vision-and-language models (VLMs) in multimodal settings involving quantified statements about images, using a novel dataset. Studying alignment between human and VLM judgments can reveal the extent to which such models grasp the semantics of natural language. The inclusion of a vision modality allows us to provide context in the form of both visual and textual information ( Zhang et al., 2024 ; Ghosh et al., 2024 ) . We examine to what extent visual cues influence state-of-the-art VLMs’ understanding and production of expressions containing vague quantifiers, and how this compares to human linguistic intuitions (Figure 1 ). By zooming in on such visual and contextual variables, our work follows the spirit of recent research exploring the grounding capabilities of VLMs ( Zellers et al., 2019 ; Thrush et al., 2022 ; Zhang et al., 2022a ; Parcalabescu et al., 2022 ; Chen et al., 2023 ; Kamath et al., 2024 ; Wang et al., 2024 , e.g.) . The contributions of this paper are as follows. • We release VAQUUM ( Va gue Qu antifiers with H um an Judgments), a new dataset pairing images of different types of objects with their counts, as well as human judgments of different quantified statements corresponding to the image. While designed for our research objectives, the dataset’s wider utility is discussed in Section 7 . • We analyze the features of the visual context that influence both human and model judgments on the appropriateness of different vague quantifiers, including counts, the segmentation area occupied by the target objects, and aspects of world knowledge such as their normative size. • We show that VLMs do, to some extent, follow human patterns in judging the appropriateness of vague quantifiers. However, the behavior of models and their degree of alignment with human judgments depend on the evaluation paradigm used (Figure 1 ): approaches that rely on extracting probabilities for quantified statement or that allow the model to choose the most appropriate statement, yield better alignment than methods which prompt the models to numerically rate the statements for their appropriateness given an image. 2 Related Work The use and judgment of vague quantifiers have been studied extensively in formal semantics and psycholinguistics. Recent years have also seen a growing but relatively limited interest in studying (V)LM behavior with linguistic quantifiers. Vague quantifiers in formal semantics Work on vague quantifiers in formal semantics sits at the crossroads of generalized-quantifier theory and degree semantics. The foundational work of Barwise and Cooper (1981) drew attention to properties of natural language quantifiers such as monotonicity and entailment; this serves as the basis for subsequent work extending generalized quantifier theory to vague quantifiers ( Partee, 1988 ; Fernando and Kamp, 1996 ; Cohen, 2001 , e.g.) . A parallel line of work has focused on the treatment of vagueness in natural language, e.g. in gradable adjectives. Here, an important insight is that vague predicates are frequently interpreted with respect to a standard that is set by some salient frame of reference (e.g. Klein, 1980 ; Graff, 2000 ; Kennedy, 2007 ; van Deemter, 2010 , i.a.) . Solt (2011) shows that comparison classes (frames of reference) also play an important role in the interpretation of vague quantifiers such as many and few . Vague quantifiers in psycholinguistics It has been suggested that humans make use of an approximate number system ( Feigenson et al., 2004 ; Dehaene, 2011 ; Coventry et al., 2005 ) , where vague terms might not refer to exact numbers but rather approximations thereof. Related to this, subitizing is the ability to instantly and accurately recognize a small number of items without going through the process of counting. Humans are generally able to do this for object counts up to 4 ( Kaufman et al., 1949 ; Mandler and Shebo, 1982 ) . Above this subitizability threshold, they tend to use quantifiers ( Barr et al., 2013 ; Berger et al., 2023 ) . However, it has also been shown that quantifier comprehension and use go beyond (approximations of) the cardinality of the targeted object. Factors include object size ( Hörmann, 1983 ; Newstead and Coventry, 2000 ) ; the number and proportions of other objects in the scene ( Coventry et al.,