On Reference (In-) Determinacy in Natural Language Inference
Sihao Chen Chaitanya Malaviya Alex Fabrikant, Hagai Taitelbaum Tal Schuster Senaka Buthpitiya Dan Roth · 2025
Read On Reference (In-) Determinacy in Natural Language Inference by Sihao Chen Chaitanya Malaviya Alex Fabrikant, Hagai Taitelbaum Tal Schuster Senaka Buthpitiya Dan Roth (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05793.txt.
Introduction
Sihao Chen and co-authors question a hidden assumption in language inference tests in this 2025 paper. Such tests ask whether a second sentence follows from a first. Builders usually assume both sentences refer to the same context, an assumption the authors call reference determinacy.
Models trained under that assumption fail when the two sentences come from different contexts, as in fact checking. The team built a diagnostic test where the first sentence is pulled from Wikipedia and may not match the second's context. Both tuned models and large models given a few examples failed to spot the mismatch. They predicted contradiction or entailment far too often. The ambiguity may also explain some human disagreement on these tests.
Catalog record
| CREATOR | Sihao Chen Chaitanya Malaviya Alex Fabrikant, Hagai Taitelbaum Tal Schuster Senaka Buthpitiya Dan Roth |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-05793
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On Reference (In-) Determinacy in Natural Language Inference Sihao Chen Chaitanya Malaviya Alex Fabrikant, Hagai Taitelbaum Tal Schuster Senaka Buthpitiya Dan Roth Abstract We revisit the reference determinacy (RD) assumption in the task of natural language inference (NLI), i.e., the premise and hypothesis are assumed to refer to the same context when human raters annotate a label. While RD is a practical assumption for constructing a new NLI dataset, we observe that current NLI models—which are typically trained solely on hypothesis-premise pairs created with the RD assumption—fail in downstream applications such as fact verification, where the input premise and hypothesis may refer to different contexts. To highlight the impact of this phenomenon in real-world use cases, we introduce RefNLI , a diagnostic benchmark for identifying reference ambiguity in NLI examples. In RefNLI , the premise is retrieved from a knowledge source (i.e. Wikipedia) and does not necessarily refer to the same context as the hypothesis. With RefNLI , we demonstrate that finetuned NLI models and few-shot prompted LLMs both fail to recognize context mismatch, leading to > 80 % >80\% false contradiction and > 50 % >50\% entailment predictions. We discover that the existence of reference ambiguity in NLI examples can in part explain the inherent human disagreements in NLI, and provide insight into how the RD assumption impacts NLI dataset creation process. https://github.com/refnli-authors/refnli 1 Introduction Natural Language Inference (NLI), or Recognizing Textual Entailment (RTE), provides a general task format for evaluating the semantic relation between two pieces of text, where a system is expected to predict if a hypothesis statement can be inferred from a given premise. For the past few decades, NLI has been the centerpiece for the development and evaluation of language understanding systems Dagan et al. 2005 ; Bowman et al. 2015 ; Williams et al. 2018 ; Nie et al. 2020a . As the use of NLI now spreads across a wider variety of downstream applications, such as text classification Yin et al. 2019 , fact verification Schuster et al. 2021 , hallucination detection Kryscinski et al. 2020 , text attribution Gao et al. 2023 , etc., it is important to understand how the definitions and assumptions made for collection of previous NLI datasets and models trained on them affect their usefulness in downstream use cases. Premise : A black race car starts up in front of a crowd of people. Hypothesis : A man is driving down a lonely road. Original Label in SNLI : Contradiction (5/5) Label (without Reference Determinacy ): Neutral Table 1: An example from the SNLI dataset Bowman et al. 2015 with all five annotators agreeing on the hypothesis contradicting the premise, under the reference determinacy assumption, i.e. the events described in the premise and hypothesis happen on the same road. Without the assumption, the label would likely be neutral. In this paper, we revisit and study the effect of reference determinacy (RD), a common assumption formed in the labeling of NLI datasets. With RD, the NLI label between a pair of premise and hypothesis is annotated under the assumption that the pair refer to the same context Bowman et al. 2015 . We illustrate the idea behind RD through an example in Table 1 , where the premise and the hypothesis describe two different events. The premise contradicts the hypothesis (i.e., premise \rightarrow\neg hypothesis) only when we opt to assume that the two events happen on the same road at the same time. Otherwise, the pair would be labeled neutral , as the two events are most likely unrelated. RD is a practical assumption for the NLI label definition. Without the RD assumption, the entailment and contradiction relations would only exist when the hypothesis and premise describe functional relations that are universally true or false Ritter et al. 2008 , e.g. factual knowledge about an entity. For this reason, most large-scale NLI benchmarks follow the RD assumption during their annotation processes ( section 2 ). However, if we train NLI models exclusively on hypothesis-premise pairs created with the RD assumption, this could lead to the resulting models having limited ability to recognize if a hypothesis is relevant to a premise. We demonstrate the trickle-down effects of such NLI model behavior in downstream tasks such as fact verification. Specifically, we sample claims from FEVER Thorne et al. 2018 and VitaminC Schuster et al. 2021 and study how NLI models behave when used to verify against evidence retrieved from the web. From the sampled claims, we construct the RefNLI benchmark ( section 3 ), which features 1,143 NLI pairs with expert judgements for whether the premise and hypothesis refer to the same context, as well as the correct NLI label. With RefNLI , we observe that both finetuned NLI models as well as LLMs few-shot prompted to classify 3-way NLI labels often fail to recognize context mismatches, which leads to many false entailment and contradiction predictions. On five popular NLI datasets ( section 4 ), we demonstrate that different combinations of training datasets result in similar type of reference (in-)determinacy problem in the finetuned model. This indicates the existence of a reference determinacy bias in all five datasets, which we discuss in the context of how each of the five datasets are created. We propose strategies to filter out entailment or contradiction examples labeled only due to the reference determinacy assumption, and show this can mitigate the reference determinacy bias of finetuned NLI models at inference time. Reference determinacy, we discover, can also partly explain part the distribution of human disagreements of NLI labels, a problem known to be widespread in popular NLI datasets Pavlick and Kwiatkowski 2019 ; Nie et al. 2020b . Our analysis shows that human typically disagree more on examples where reference determinacy cannot be safely assumed, and disagreements happen when annotators are instructed to do so regardless. In summary, our contributions in the paper are: • We introduce the RefNLI benchmark, a dataset featuring 1,143 examples for studying the the effect of reference determinacy in NLI, a common assumption in the creation processes of NLI datasets. • With RefNLI , we investigate the downstream impact of the reference determinacy assumption of NLI dataset creation process. We show that finetuned NLI models and LLMs exhibit reference determinacy bias and often fail to recognize context mismatches. • We discover and study the connection of the reference determinacy assumption to the inherent human disagreement on NLI labels. 2 The Reference Determinacy Assumption When we create and label NLI examples, reference determinacy (RD) is a practical assumption for guaranteeing the correctness and consistency of annotated labels. For instance, suppose a hypothesis and premise pair both mention John Doe , the perceived entailment or contradiction relation could change based on whether we believe the two “John Doe”s are a single real-world person. The creation processes of most NLI datasets assume reference determinacy. For example, in SNLI Bowman et al. 2015 and MNLI Williams et al. 2018 , annotators were asked to write novel hypotheses that are either true/false/neutral in the context of a given premise. During labeling, the hypothesis is interpreted in the context of the premise, where entities and events in the two are assumed to be co-refer between the