Generics are puzzling. Can language models find the missing piece?
Gustavo Cilleruelo Calderón aaa Emily Allaway aaa Barry Haddow aaa Alexandra Birch · 2024
Read Generics are puzzling. Can language models find the missing piece? by Gustavo Cilleruelo Calderón aaa Emily Allaway aaa Barry Haddow aaa Alexandra Birch (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11318.txt.
Introduction
Gustavo Cilleruelo Calderón and co-authors ask how language models handle broad claims about groups in work from 2024. Sentences like birds fly state a general rule with no stated share. People bend such claims with context in subtle ways.
The team built a data set of 2,873 sentences taken from real text, each shown in context. They define a score based on how surprised a model is by each sentence. Models treated broad claims as more swayed by context than claims with words like all or some. About a fifth of the broad claims stated weak rules that hold only rarely. The study also shows how bias about social groups can show up in model scores.
Catalog record
| CREATOR | Gustavo Cilleruelo Calderón aaa Emily Allaway aaa Barry Haddow aaa Alexandra Birch |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-11318
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generics are puzzling. Can language models find the missing piece? Gustavo Cilleruelo Calderón aaa Emily Allaway aaa Barry Haddow aaa Alexandra Birch Abstract Generic sentences express generalisations about the world without explicit quantification. Although generics are central to everyday communication, building a precise semantic framework has proven difficult, in part because speakers use generics to generalise properties with widely different statistical prevalence. In this work, we study the implicit quantification and context-sensitivity of generics by leveraging language models as models of language. We create ConGen , a dataset of 2873 naturally occurring generic and quantified sentences in context, and define p-acceptability , a metric based on surprisal that is sensitive to quantification. Our experiments show generics are more context-sensitive than determiner quantifiers and about 20% of naturally occurring generics we analyze express weak generalisations. We also explore how human biases in stereotypes can be observed in language models 1 1 1 Code and data are available in https://github.com/ilyocoris/generics_are_puzzling . 1 Introduction Humans use generalisations to abstract away from particular objects, events or facts and convey regularities about the world. In this work, our focus is on generic sentences, such as insects have six legs or mosquitoes carry malaria , which express generalisations without explicit quantification. These two generic sentences are acceptable in many contexts, but the quantifications they convey are widely different: almost all insects have six legs, but fewer than 1% of mosquitoes carry malaria. One way of expressing generalisations in language is through explicitly quantified sentences, such as most insects are nocturnal or some mosquitoes have white stripes . Quantified sentences express statistical claims about the members of a kind that share the predicated property: for example most if a majority of insects are nocturnal or some if a minority of mosquitoes have white stripes. Even as generics seem to express inconsistent quantifications, they are at the heart of communication and dissemination in science ( DeJesus et al., 2019 ; Bowker, 2022 ) , medical research ( Peters et al., 2024 ) , and politics ( Novoa et al., 2023 ) . Furthermore, in the social realm generics serve as linguistic vehicle for social essentialism ( Rhodes et al., 2012 ) and stereotyping ( Leslie, 2017 ; Bosse, 2022 ) . The nature of generics and their importance in communication has led to extensive literature on the semantics of generic sentences ( Carlson, 1977b ; Cohen, 1999 ; Leslie, 2008 ; Liebesman, 2011 ; Sterken, 2015a ; Nickel, 2016 ; Tessler and Goodman, 2016 ; Stovall, 2019 ; Nguyen, 2020 ; Bosse, 2021 ; Kirkpatrick, 2023 , e.g.,) . However, many open questions remain. These include how they relate to quantifiers and the degree to which generics are context sensitive. In this work, we use language models to explore the implicit quantification and context-sensitivity of generics, and how they are affected by human biases around stereotypes. Language models have demonstrated unprecedented performance in a variety of linguistic tasks, such as machine translation ( Kocmi et al., 2024 ) or conversational assistance ( Chiang et al., 2024 ) . We describe how speakers use generics by studying the surprisal in language models for various naturally occurring generic and quantified sentences. Most existing datasets of generics are synthetic, often derived from knowledge bases or generated by language models ( Bhakthavatsalam et al., 2020 ; Allaway et al., 2024 ) . Since the examples in these datasets are machine-generated and/or lack a context in which they might be uttered, there is no guarantee that they represent how speakers actually use generics. Therefore, in this work we introduce ConGen , a dataset of naturally occurring generic and quantified sentences with contexts. In order to study generics and quantification, we define the p-acceptability metric; given a set of quantifiers, it indentifies the one that best fits a sentence by using the surprisal of a language model. Previous work either fails to use surprisal to describe quantification or focuses on prompting ( Collacciani et al., 2024 ; Allaway et al., 2024 ) . We validate this metric by showing that it recovers the expected dynamics of quantifiers ( all , most , some ) and the generic on two datasets of generics ( ConGen and GenericsKB ). We then use our metric to study different aspects of generics. Our contributions are ( i ) ConGen , a dataset of naturally occurring bare plural generic and quantified sentences in contexts, ( ii ) p-acceptability, a new metric based on language models that is sensitive to quantification and ( iii ) insights into how generics are used, including weak generalisations, context-sensitivity and stereotypes. 2 Background Semantic theories of generics guide and scaffold our experimental design. In what follows, we present linguistic background on generics (§ 2.1 ) and then we sketch two theories of genericity related to our experiments (§ 2.2 ): generics-as-defaults and contextualism . Finally, we introduce two phenomena that involve generics (§ 2.3 ): stereotypes and generic overgeneralisation . These theoretical elements motivate our research questions and frame the interpretation of experimental results. 2.1 Generics The term generics covers multiple linguistic phenomena that abstract away from particular objects, members, or events. In our work, we focus on one specific kind of generics: bare plural characteristic sentences . Bare plurals are noun phrases in plural form without a definite or indefinite article 2 2 2 Sharks attack bathers is a bare plural generic. The same generic could also be expressed in English with the definite ( the shark attacks bathers ) or indefinite ( a shark attacks bathers ) articles. . Characteristic sentences are propositions that do not express specific episodes or isolated facts, but rather report a kind of general property or regularity ( Carlson and Pelletier, 1995 ) . In linguistics, generics are traditionally analyzed as quantifiers ( Carlson, 1977b ) , with an unpronounced implicit operator Gen that has a role similar to that of most or generally in explicit quantification ( Lewis, 1975 ) . However, there is no consensus on what the semantic content of Gen is, how it is determined or even if it exists ( Carlson and Pelletier, 1995 , cf.) . Despite this, most real-life generics are majority generics : they are acceptable when a majority of the members of the kind in question satisfies the predicated property (e.g. ravens are black ). However, some generics, such as mosquitoes carry malaria , ducks lay eggs , or bees reproduce , are often used even though they express statistically weak generalisations. We call these weak generics ( Almotahari, 2022 ) . Weak generics are broadly categorized into two types ( Leslie, 2007 ) : those that express properties that are characteristic of the kind, but are only possessed by a minority of its members ( minority generics ) and those that express dangerous, striking or appalling characteristics ( striking generics ). In addition to the above, Leslie et al. (2011) also distinguish quasi-definitional generics. These predicate a property true of all the members of the kind, without exceptions (e.g. beetles are insects ). 2.2 Generics in Philosophy of language In this section we introduce two influential accounts from philosophy of language: generics-as-defaults and contextualism. These supply contrasting