Reproducing NevIR: Negation in Neural Information Retrieval

Coen van den Elsen, Francien Barkhof, Thijmen Nijdam, Simon Lupart, Mohammad Aliannejadi · 2025

Read Reproducing NevIR: Negation in Neural Information Retrieval by Coen van den Elsen, Francien Barkhof, Thijmen Nijdam, Simon Lupart, Mohammad Aliannejadi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13506.txt.

Introduction

Coen van den Elsen and co-authors repeat and extend, in a 2025 paper, a study on how search engines handle the word not. Negation is basic in human speech, yet neural search tools lean on language models that may gloss over it. The earlier NevIR study found most models rank negated queries at or below chance.

The team reran the first tests and added fresh state-of-the-art search models. A new class of large model re-rankers, which sort whole lists at once, beat the rest but still trailed human skill. A second test set, built for exclusion queries, told a like story. Training on one set did not reliably lift scores on the other. Only two model types held up across both tasks with fair results.

Catalog record

CREATORCoen van den Elsen, Francien Barkhof, Thijmen Nijdam, Simon Lupart, Mohammad Aliannejadi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13506

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reproducing NevIR: Negation in Neural Information Retrieval Coen van den Elsen, Francien Barkhof, Thijmen Nijdam, Simon Lupart, Mohammad Aliannejadi Abstract . Negation is a fundamental aspect of human communication, yet it remains a challenge for Language Models (LMs) in Information Retrieval (IR). Despite the heavy reliance of modern neural IR systems on LMs, little attention has been given to their handling of negation. In this study, we reproduce and extend the findings of NevIR, a benchmark study that revealed most IR models perform at or below the level of random ranking when dealing with negation. We replicate NevIR’s original experiments and evaluate newly developed state-of-the-art IR models. Our findings show that a recently emerging category—listwise Large Language Model (LLM) re-rankers—outperforms other models but still underperforms human performance. Additionally, we leverage ExcluIR, a benchmark dataset designed for exclusionary queries with extensive negation, to assess the generalisability of negation understanding. Our findings suggest that fine-tuning on one dataset does not reliably improve performance on the other, indicating notable differences in their data distributions. Furthermore, we observe that only cross-encoders and listwise LLM re-rankers achieve reasonable performance across both negation tasks. *[inlinelist,1]label=),itemjoin=, ,itemjoin*=, and Conference: Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 13–18, 2025; Padua, Italy Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’25), July 13–18, 2025, Padua, Italy DOI: 10.1145/3726302.3730294 ISBN: 979-8-4007-1592-1/2025/07 CCS: Information systems Information retrieval CCS: Information systems Evaluation of retrieval results CCS: Information systems Test collections † † cc-license: by 1. Introduction Negation plays a crucial role in human communication, enabling the expression of contradictions, exclusions, and oppositions. Despite its common use in natural language, effectively handling negation remains a significant challenge for Language Models (LMs) ( Ravichander et al., 2022 ) . This issue has notable implications for Information Retrieval (IR) tasks, as different and complex forms of negation can appear in both user queries and documents ( Ravichander et al., 2022 ) . This misinterpretation of negation is undesirable and can be harmful, particularly in high-risk retrieval tasks such as healthcare or legal settings, where it can lead to serious errors ( Weller et al., 2024b ) . For example, as shown in Figure 1 , misinterpreting negation in the medical question-answering domain can have severe and even catastrophic consequences. As can be seen in Figure 1 , missing the negation on the “non-allergic” cases would lead to a catastrophic recommendation generated by the system that prescribes epinephrine to the user with no peanuts allergy. Figure 1. This example illustrates how misinterpreting a negation in a medical context could cause someone to misuse a critical emergency medication like epinephrine Prior work on negation in IR, primarily from the pre-neural era, relied on rule-based and early statistical methods ( Chapman et al., 2001 ; Harkema et al., 2009 ; Goldin and Chapman, 2003 ; Uzuner et al., 2011 ) , focusing on explicit lexical cues. However, these methods largely overlooked more nuanced or implicit forms of negation. Despite its importance, there has been little research on how modern neural IR models handle negation. To address this gap, Weller et al. (2024b) propose a novel benchmark dataset —NevIR— to specifically evaluate how more recent neural IR systems handle negation. The NevIR dataset consists of pairs of queries, each semantically aligned with exactly one of two contrastive documents. These document pairs differ only in the presence or absence of negation terms (e.g., “The box is locked.” vs. “The box is unlocked.”). The authors tested neural IR models by assessing their ability to correctly rank the appropriate document above its negated counterpart. The results showed that most state-of-the-art (SotA) IR models struggle with negation, often performing worse than random ranking. Given these limitations identified by the NevIR benchmark, an important question arises: do these findings generalise to newer IR model families, particularly LLM-based IR methods? Recently, LLMs have demonstrated strong effectiveness in IR, making them promising candidates for retrieval tasks ( Ma et al., 2024 ; Sun et al., 2023 ; Qin et al., 2024 ; Ma et al., 2023 ) , but their ability to process negation remains uncertain. In this work, we therefore not only reproduce the experiments of the NevIR paper ( Weller et al., 2024b ) , but also extend its evaluation to LLM-based retrieval methods. In doing so, we aim to assess whether recent advances in IR have improved negation handling or whether it remains a persistent challenge. Furthermore, Weller et al. (2024b) propose a preliminary solution for handling negation by fine-tuning IR models on the NevIR dataset. While this improves performance significantly, it remains unclear whether models truly learn negation patterns or overfit the data distribution of the NevIR dataset. This is particularly interesting given that the dataset is created synthetically and could potentially lack negation and language diversity. To address this, we build on the recent work of Zhang et al. (2024) , who introduced ExcluIR—a benchmark focused specifically on exclusionary queries. For example, the query “What other flowers besides tulips grow in the Netherlands?” expresses exclusion by implicitly negating one category. By fine-tuning and evaluating models on both NevIR and ExcluIR, we explore the generalisation of negation handling across datasets and assess the effectiveness of NevIR in helping models understand negation in IR. Through this study, we seek to extend the current work on negation in IR, contributing to a clearer understanding of how different datasets and models affect negation understanding. Additionally, we aim to extend the initial NevIR experiments by exploring the potential trade-offs between improving negation understanding and preserving ranking performance. We address the following research questions: RQ1 How do the negation understanding results of the NevIR dataset generalise to the most recent retrieval architectures, such as listwise LLM re-rankers? RQ2 How do listwise LLM re-rankers perform when fine-tuned on the negation data of NevIR? RQ3 How does fine-tuning on other negation datasets, such as ExcluIR, improve generalisability on NevIR? RQ4 How can performance on a negation dataset be improved while maintaining strong ranking capabilities? In response to these questions, we present the following findings: • We observe a positive trend in handling negation on newly developed SotA IR models. Notably, a newly emerged model category, listwise LLM re-rankers, achieved 20% gains compared to the previous approaches. We also observe that the recent advances in cross-encoders and bi-encoders with models such as GritLM and jina-reranker-v2-base-multilingual show an improvement in negation understanding. • When fine-tuning, we find that, despite its superior base performance, listwise LLM re-rankers have similar learning capabilities for the negation task as other model categories. • Fine-tuning experiments on NevIR and ExcluIR reveal that only the model from the cross-encoder category can generalise between the two negation datasets. • Our analysis of training IR models on negation data shows that a trade-off selection criterion can help mitigate overfitting. Our code, which ensures full reproducibility of our experiments, is publicly available on GitHub 1 1 1 https://github.com/thijmennijdam/NevIR-reproducibility . 2. Related Work Negation in NLP. Negation is a complex semantic phenomenon in natural language that affects the performance of various Natural Language Processing (NLP) applications ( Sineva et al., 2021 ; Morante and

Read the complete text