Let’s CONFER: A Dataset for Evaluating Natural Language Inference Models on CONditional InFERence and Presupposition
Tara Azin, Daniel Dumitrescu, Diana Inkpen, Raj Singh · 2025
Read Let’s CONFER: A Dataset for Evaluating Natural Language Inference Models on CONditional InFERence and Presupposition by Tara Azin, Daniel Dumitrescu, Diana Inkpen, Raj Singh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06133.txt.
Introduction
Tara Azin, Daniel Dumitrescu, Diana Inkpen, and Raj Singh build in 2025 a test set for how well AI models handle hidden assumptions in if-then sentences. Language models must judge whether one sentence follows from another. They often stumble when a claim is assumed rather than stated outright.
Their set, CONFER, pairs each question with three answers and three images that split text-based guesses from true checks. Four inference models and several large chat models face the set, with and without hints. All models struggle with assumed meaning in if-then forms. Extra training on older sets does not fix the gap, which marks a clear weak spot for builders to address.
Catalog record
| CREATOR | Tara Azin, Daniel Dumitrescu, Diana Inkpen, Raj Singh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06133
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Let’s CONFER: A Dataset for Evaluating Natural Language Inference Models on CONditional InFERence and Presupposition Thanks: Citation : This paper is published in the Proceedings of the 38th Canadian Conference on Artificial Intelligence (CAIAC 2025). Please cite the conference version at https://caiac.pubpub.org/pub/keh8ij01 Tara Azin, Daniel Dumitrescu, Diana Inkpen, Raj Singh Abstract Natural Language Inference (NLI) is the task of determining whether a sentence pair represents entailment, contradiction, or a neutral relationship. While NLI models perform well on many inference tasks, their ability to handle fine-grained pragmatic inferences, particularly presupposition in conditionals, remains underexplored. In this study, we introduce CONFER, a novel dataset designed to evaluate how NLI models process inference in conditional sentences. We assess the performance of four NLI models, including two pre-trained models, to examine their generalization to conditional reasoning. Additionally, we evaluate Large Language Models (LLMs), including GPT-4o, LLaMA, Gemma, and DeepSeek-R1, in zero-shot and few-shot prompting settings to analyze their ability to infer presuppositions with and without prior context. Our findings indicate that NLI models struggle with presuppositional reasoning in conditionals, and fine-tuning on existing NLI datasets does not necessarily improve their performance. K eywords natural language inference \cdot pragmatic inference \cdot presupposition \cdot conditional reasoning \cdot large language models 1 Introduction Inferring whether a pair of sentences represents an entailment, a contradiction, or a neutral relation is fundamental to Natural Language Inference (NLI). This task has proven to be effective in training and evaluating models on textual reasoning. However, not all forms of inference are handled equally well by NLI models. Among different types of inference, presupposition is a specifically interesting case. Presupposition is a form of pragmatic inference that relies on shared assumptions between speakers rather than purely on logical entailment. It is different from entailment in that it is not canceled when placed in entailment-canceling environments (e.g., negation, questions, modal verbs). Table 1 shows the distinction between presupposition and entailment under negation, based on standard linguistic theories of presupposition projection [ 1 , 2 ] . Presuppositions become even more complex when embedded in conditional sentences, where the presupposition inherited by the whole sentence appears to be less predictable. Unlike in simple declarative sentences, where presuppositions tend to persist, conditionals introduce dependencies that can sometimes modify the presupposition of an embedded constituent. In particular, sentences in a conditional format like “ \text{If }A,\text{then}\ B_{p} ”, where p is the presupposition of the consequent (B), appear to presuppose either p itself or “If A then p” (further details will be discussed in Section 2 ). This variability complicates how presuppositions are interpreted and accommodated in discourse. Understanding how presuppositions behave in such structures is essential for natural language processing tasks, since models must distinguish between context-dependent inferences and inferences that hold unconditionally. Sentence: The chocolate cake is delicious. Negated Sentence: The chocolate cake is not delicious. Presupposition Entailment Implied Meaning There is a cake. Something is delicious. Remains True After Negation? Yes No Table 1: Distinction between presupposition and entailment under negation. The presupposition of a sentence, such as the existence of a chocolate cake , remains true even when the sentence is negated. In contrast, the entailment, which follows from the truth condition of the original proposition does not survive negation, and the entailment is canceled. Returning to the definition and function of NLI models, since presupposition involves a specific type of inference that interacts with both semantic and pragmatic reasoning, NLI provides a structured framework to evaluate how well models handle such inferences. It remains unclear whether NLI-trained models truly represent presuppositional reasoning or simply learn surface-level correlations. Therefore, focusing on a specific type of pragmatic reasoning within a fixed linguistic structure could be an effective approach to evaluate a model’s ability to handle complex inference tasks. To investigate this, we introduce CONFER, a dataset that is specifically designed to examine machine understanding of presuppositions in certain types of conditional sentences 1 1 1 https://github.com/Conditional-NLI/CONFER . Our dataset consists of 18,000 sentence pairs, generated semi-automatically using linguist-designed templates, allowing us to create a sizable, lexically diverse, and carefully controlled dataset that focuses on specific types of conditional sentences. To our knowledge, no existing NLI dataset is specifically dedicated to conditional sentences. Other available datasets [ 3 , 4 ] contain either no conditionals or only a limited number, making their evaluation of such structures less comprehensive. As part of our study, we evaluated two pre-trained models, RoBERTa and DeBERTa, using our data alongside two other presupposition datasets, IMPPRES [ 3 ] and NOPE [ 4 ] to assess their performance on pragmatic inference tasks. We also tested four Large Language Models (LLMs): Gemma, Llama, GPT-4o, and DeepSeek-R1, in both zero-shot and few-shot prompting settings to examine their ability to understand specific types of conditionals in contexts with and without prior training. Our findings show that NLI models, including LLMs, struggle to generalize to complex conditional structures, particularly in presuppositional reasoning. While fine-tuning on standard datasets enhances performance on inference tasks, it does not transfer well to conditionals and often results in performance degradation. Experiments with fine-tuning and prompting techniques indicate that the models have limitations in handling the complexities of conditional reasoning. These results emphasize the need for more diverse and structured datasets to enhance model performance in pragmatic inference. Our contributions are as follows: • We introduce Confer , a dataset designed to evaluate pragmatic inference in conditionals. • To the best of our knowledge, this is the first study to evaluate pre-trained NLI models, as well as large language models (LLMs), on presupposition projection in conditional inference. • The dataset, prompts, code, and experimental results are publicly released to facilitate further research in pragmatic and conditional inference within NLI. 2 Background Pragmatic Inference. Pragmatic inference extends beyond logical entailment by taking into account contextual and speaker-driven reasoning. It refers to a relationship between two sentences that depends on the context of utterance and the conversational objectives of the speakers. Unlike semantic entailment, which is determined by the logical relationship between sentence meanings in isolation, pragmatic inference results from how language is used in communication and how interlocutors interpret meaning beyond explicit content [ 2 ] . Among various types of pragmatic inference, presuppositional inference is particularly complex because it interacts with both semantic structure and pragmatic accommodation. A more detailed discussion of presupposition follows below. Presupposition. Presupposition is the background assumption that a speaker considers to be true when making an utterance. In semantics, presupposition is typically viewed as a relationship between two sentences, where the truth of one sentence is a necessary condition for the other to have a truth value [ 5 ] . Presuppositions are typically associated with specific linguistic expressions, known as presupposition triggers, which signal their presence in a sentence. These triggers include iterative adverbs (e.g., again, anymore), factive verbs (e.g., realize, know), definite noun phrases (e.g., the king, her book), and change-of-state verbs (e.g., break, crack). For example, in the sentence Mary failed the exam again , the iterative adverb again signals that Mary has failed the exam before [ 6 ] . Similarly, in John regrets quitting his job , the factive verb regrets presupposes that John quit his job. Unlike entailments, presuppositions