Meaning Beyond Truth Conditions: Evaluating Discourse Level Understanding via Anaphora Accessibility
Xiaomeng Zhu Zhenghao Zhou Simon Charlow Robert FrankDepartment of LinguisticsYale University {miranda.zhu, herbert.zhou, simon.charlow, robert.frank}@yale.edu · 2025
Read Meaning Beyond Truth Conditions: Evaluating Discourse Level Understanding via Anaphora Accessibility by Xiaomeng Zhu Zhenghao Zhou Simon Charlow Robert FrankDepartment of LinguisticsYale University {miranda.zhu, herbert.zhou, simon.charlow, robert.frank}@yale.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-14119.txt.
Introduction
Xiaomeng Zhu, Zhenghao Zhou, Simon Charlow, and Robert Frank wrote a 2025 paper on how to test whether a language model grasps a text as a whole, not just line by line. Word meaning and sentence meaning are the old test levels. A whole text is more: words in one sentence point back to things named in earlier sentences, and those links follow rules.
The team built a test set from dynamic semantics, a known branch of formal language study that sets out those rules. One task asks which earlier thing a backward-pointing word, an anaphor, can reach. Humans and models took the same tests. On some tasks, the two matched. On others, they diverged. The divergence, the team says, comes from how models lean on specific words, while humans lean on the structure of the text.
Catalog record
| CREATOR | Xiaomeng Zhu Zhenghao Zhou Simon Charlow Robert FrankDepartment of LinguisticsYale University {miranda.zhu, herbert.zhou, simon.charlow, robert.frank}@yale.edu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-14119
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Meaning Beyond Truth Conditions: Evaluating Discourse Level Understanding via Anaphora Accessibility Xiaomeng Zhu Zhenghao Zhou Simon Charlow Robert FrankDepartment of LinguisticsYale University {miranda.zhu, herbert.zhou, simon.charlow, robert.frank}@yale.edu Abstract We present a hierarchy of natural language understanding abilities and argue for the importance of moving beyond assessments of understanding at the lexical and sentence levels to the discourse level. We propose the task of anaphora accessibility as a diagnostic for assessing discourse understanding, and to this end, present an evaluation dataset inspired by theoretical research in dynamic semantics. We evaluate human and LLM performance on our dataset and find that LLMs and humans align on some tasks and diverge on others. Such divergence can be explained by LLMs’ reliance on specific lexical items during language comprehension, in contrast to human sensitivity to structural abstractions. * * footnotetext: Equal contribution. 1 Introduction The success of modern large language models (LLMs) depends on their capacity for natural language understanding (NLU), i.e., the ability to extract the semantic information contained in a text. Systematic assessment of NLU abilities has been carried out using a diverse set of evaluation tasks, but few of them target whether LLMs accurately represent and update states of natural language discourse. Successful interpretation of discourse requires the ability to use pronominal expressions to refer to entities that have been introduced in a text. The felicity of pronominal anaphora , i.e., using pronouns to refer back to discourse referents introduced earlier, is influenced by the semantic scope of the antecedent: \pex [aboveexskip=4pt, belowexskip=4pt]<demo> {A, #Every} farmer worked in his field. He dreamed of the harvest. \xe Example ( \getref demo) shows that an entity introduced by an existential quantifier is accessible in the same sentence, as well as in subsequent sentences. In contrast, entities introduced by universal quantifiers are only accessible to pronouns in the same sentence; anaphora is infelicitous otherwise. This is illustrated in Figure 1 : the discourse referent is subordinated to the universal quantifier — that is, inaccessible outside its scope, which extends to the end of the first sentence in the sequence. This makes subsequent reference to he in the second sentence infelicitous. The process of introducing discourse referents is formalized in ‘dynamic’ variants of formal semantics (e.g., Heim 1983 ; Groenendijk and Stokhof 1991 ; Kamp et al. 2010 ). In dynamic semantics, utterances precipitate changes in the discourse state, for example by introducing discourse referents. This gives rise to notions of discourse or textual scope which differentiate (e.g.) existential and universal quantifiers, in line with Figure 1 . Figure 1: Quantifier scope and its impact on anaphora. Here, we focus on one aspect of discourse-level semantic knowledge, namely the fine-grained interactions between semantic scope and referent accessibility. We investigate whether LLMs demonstrate knowledge of the semantic scope properties of various quantifiers and logical connectives, and whether this knowledge is used to generate and update representations of discourse states in human-like ways. Figure 2: Proposed hierarchy of levels of semantic understanding abilities. Contribution We make the following contributions: • In Section 2 , we propose a hierarchy of levels of semantic understanding abilities, which can serve as a guideline for characterizing the kinds of semantic knowledge that LLMs have. • In Section 3 , we propose an evaluation dataset covering discourse anaphora across a variety of linguistic constructions, all of which require sensitivity to the way in which the form of language determines the ways discourse states are implicitly updated in natural discourse. • In Sections 4 through 6 , we evaluate both LLMs and humans with our dataset, and uncover intriguing patterns where human and model behavior align and differ. 2 Levels of Semantic Understanding Figure 2 illustrates three different levels of natural language understanding: (i) lexical level, (ii) sentential level, and (iii) discourse level. Semantic competence, we propose, requires knowledge of all of these. We discuss each one in detail and review existing work that has tried to evaluate LLM capacities at that level. 2.1 Lexical Level We define lexical level understanding as knowing the meaning of individual lexical items . This requires knowledge of a word’s extension (the objects in the world that a word picks out) and its intension (the objects it would pick out if the world were different). Such knowledge allows a competent speaker to make judgments of synonymy, antonymy, entailment and the like. In LLMs, lexical knowledge corresponds to vector representations of individual tokens. Moskvoretskii et al. 2024 summarize a range of Natural Language Understanding (NLU) tasks that assess lexical level understanding: Word Sense Disambiguation, Hypernym Discovery, Taxonomy Construction, Lexical Entailment, etc. Another test of lexical semantic understanding derives from the analogical reasoning tests explored by Mikolov et al. 2013 , where word meaning is needed to complete analogies such as man : king as woman : X . All of these tasks rely on knowledge of word meaning that is independent of the effects on meaning that derives from the composition of words in phrases and sentences. 2.2 Sentence Level On top of the building blocks provided by lexical understanding, sentence understanding is the ability to integrate lexical meanings in phrases and to form coherent semantic representations for sentences . Traditionally, sentence-level meaning is identified with truth conditions and encoded using a logical formalism with rigorously defined semantics (e.g., Heim and Kratzer 1998 ). A model’s capacity to encode the truth conditions of single sentences is implicated in important NLU tasks such as Natural Language Inference (NLI), which requires LLMs to form accurate meaning representations for two sentences and classify their logical relations as entailment, contradiction, or neutral ( Williams et al. 2018 ) . Similar evaluation tasks have been created for pragmatic inferences, targeting implicature and presupposition ( Jeretic et al. 2020 ) . These works investigate meaning representations of pairs of minimally different sentences, either with respect to logical relations or pragmatic relations, without the need to connect the two sentences in sequential order or track changes at the discourse level. Another type of work at the sentence level involves ambiguities, such as scope ambiguity (e.g., Kamath et al. 2024 ): a single sentence with multiple quantifiers might allow different interpretations given specific scopal arrangements between the quantifiers. 2.3 Discourse Level We define discourse level understanding as the ability to integrate the meaning of consecutive sentences into a unified discourse representation . Discourse-level meaning requires moving beyond formalisms that express meaning as a static representation of truth conditions to dynamic formalisms in which meaning accrues via update to a contextual representation or state. One type of task that probes discourse level understanding is discourse parsing (e.g., Maekawa et al. 2024 ), which evaluates the ability of a model to determine the relationships between sentences, such as elaboration , attribution , etc. While informative, this task requires the adoption of specific assumptions about the structure and categories that determine discourse relations. An alternative, more theory-neutral evaluation considers the accumulation of information through a discourse. Li et al. 2021 examine the tracking of the state of individuals and situations across a text. They probed the internal representations of encoder-decoder transformers and found localizable, interpretable structures, supporting the claim that pretrained language models implicitly simulate entity tracking processes dynamically. Kim