Evidence Absence Is Not Evidence Insufficiency: Diagnosing NEI Construction Artifacts in Fact Verification
Jingxi Qiu, Zeyu Han, Cheng Huang · 2026
Read Evidence Absence Is Not Evidence Insufficiency: Diagnosing NEI Construction Artifacts in Fact Verification by Jingxi Qiu, Zeyu Han, Cheng Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26663.txt.
Introduction
Jingxi Qiu, Zeyu Han, and Cheng Huang diagnose why insufficient-evidence verdicts mislead in fact checking. The NEI label often comes from built evidence setups, and that setup quietly shapes what a checker learns. Their NEI-CAP protocol tags each example with its setup family, hunts shortcut cues, checks hard cases by hand, and tests transfer across setups. On SciFact with FEVER and HoVer controls, skill trained on shortcut-prone setups fails on related but insufficient evidence. Mixed training narrows but never closes the gap.
Catalog record
| CREATOR | Jingxi Qiu, Zeyu Han, Cheng Huang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73819 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.26663
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evidence Absence Is Not Evidence Insufficiency: Diagnosing NEI Construction Artifacts in Fact Verification Jingxi Qiu, Zeyu Han, Cheng Huang Abstract Evidence absence is not evidence insufficiency, but fact verification benchmarks can make them observationally similar. The Not Enough Information ( NEI ) label is often operationalized through different evidence conditions, and that choice silently determines what a verifier learns, and what its score can hide. We introduce NEI-CAP, a construction-aware diagnostic protocol for insufficient-evidence evaluation. Each NEI example carries the construction family that produced it; NEI-CAP audits shortcut cues, validates hard cases through human adjudication, and tests whether competence transfers across constructions. We instantiate the protocol in SciFact-style scientific verification, with FEVER and HoVer as bounded external controls. Across these settings, NEI competence does not transfer reliably: models trained on shortcut-prone constructions fail to recognize semantically related insufficient evidence, and mixed-construction training narrows but does not close the gap. Fixed-claim diagnostics further show that the evidence condition shifts confidence in the reference Support / Refute label, not only NEI recall, so an aggregate NEI score can hide which problem a model has actually solved. 1 Introduction A fact verification system labels a claim as supported, refuted, or Not Enough Information (NEI) when the available evidence is inconclusive ( 24 ; 28 ; 12 ) . The NEI label is meant to be evidence-conditioned: for a claim c and an evidence set E , it means that E does not establish c either way. Building those negative evidence sets, however, is itself a design step that no formal definition covers. An empty field, an off-topic passage, a high-overlap retrieval miss, and a non-rationale sentence drawn from a cited document can all be labelled NEI, and a verifier trained on one of these constructions can predict NEI for reasons that have little to do with whether the evidence is actually sufficient. Prior work on fact-verification artifacts has focused on the claim side. 23 show that FEVER can be partially solved by claim-only classifiers, and adversarial or contrastive verification resources further show that standard evidence-aware accuracy can hide brittle decision rules ( 25 ; 22 ) . A separate line studies evidence sufficiency by removing parts of otherwise-valid evidence ( 2 ; 26 ) . We study a complementary failure mode—how the negative evidence condition is built in the first place—and argue that this construction silently determines what a verifier learns and what an aggregate NEI-F1 can hide. Figure 1 illustrates the mechanism. Easy NEI can be solved by recognizing absence, format, or topic mismatch; hard NEI keeps the evidence related to the claim but incomplete, which can induce false support when the model mistakes overlap for sufficiency. NEI-CAP targets this gap by making construction family part of the evaluation record rather than treating NEI as a construction-free class. Figure 1: Conceptual illustration of NEI construction artifacts. Easy NEI constructions, such as placeholders or unrelated passages, can teach absence, format, or topic-mismatch shortcuts. Hard NEI keeps the evidence semantically related but incomplete; a verifier may therefore overpredict Support from overlap rather than recognize insufficiency. NEI-CAP records the construction family, audits shortcuts, validates hard examples, and stress-tests whether NEI competence transfers. Examples are schematic; experiments use the SciFact, FEVER, and HoVer constructions described in Sections 3 – 4 . NEI-CAP makes the construction explicit. Each NEI example carries the family of evidence condition that produced it, which lets us audit each family for shortcut features and stress-test whether a model trained on one family recognizes insufficiency in another. We instantiate the protocol on SciFact-style scientific verification ( 28 ) , with FEVER ( 24 ) , HoVer ( 12 ) , and the broader fact-verification literature as context ( 3 ; 1 ) . The headline finding is a transfer failure that aggregate NEI-F1 cannot detect. A DeBERTa verifier trained on placeholder NEI reaches perfect matched-placeholder NEI-F1 across five seeds, yet scores zero NEI-F1 on BM25 near-miss and cited non-rationale evaluation; the collapse replicates on RoBERTa and SciBERT. Probability mass shifts to Support and Refute rather than to NEI, so the failure is not a calibration artifact. Training on random-irrelevant NEI fares only marginally better, showing that the problem extends beyond placeholder detection. Mixed-construction training narrows but does not close the gap. A fixed-claim diagnostic further shows that swapping the evidence shifts confidence in the reference Support or Refute label, not only NEI recall. The construction choice therefore affects the verifier on the full three-way task, not only the NEI corner. We make three contributions. First , we recast NEI as a construction-sensitive evidence condition rather than a single negative label. Second , we introduce NEI-CAP: a diagnostic protocol that treats the construction family as an explicit evaluation variable, audits its shortcut surface, and validates hard cases through human adjudication. Third , our SciFact, FEVER, and HoVer experiments show that shortcut-prone training fails to transfer to semantically related insufficient evidence, and that multi-seed and mixed-construction protocols do not remove the need for construction-stratified reporting. 2 Related Work 2.1 Fact Verification Benchmarks Fact verification is commonly formulated as a three-way classification problem: given a claim and an evidence set, predict whether the evidence supports, refutes, or is insufficient to verify the claim. FEVER introduced a large-scale Wikipedia benchmark with Supported , Refuted , and NotEnoughInfo labels ( 24 ) ; SciFact extended the formulation to expert-written scientific claims that require retrieving evidence-containing abstracts and rationales ( 28 ) ; and HoVer added many-hop evidence retrieval, where verification can depend on facts spread across multiple Wikipedia articles ( 12 ) . MultiFC broadens fact-checking to real-world multi-domain claims ( 3 ) ; HealthFC focuses on evidence-backed medical claims ( 27 ) ; FEVEROUS adds structured table evidence ( 1 ) ; and VitaminC creates contrastive claim–evidence pairs that require sensitivity to small factual changes ( 22 ) . Rationale-centered resources further ask whether systems identify the evidence used to support predictions ( 5 ) . Across these benchmarks the NEI label is treated as a fixed third class, but how its evidence side is built is left to each benchmark’s discretion—and that choice is not made part of the evaluation protocol. 2.2 Dataset Artifacts and Behavioral Evaluation NLP benchmarks routinely contain artifacts that let models score well without learning the intended capability. In natural language inference, hypothesis-only classifiers can recover the label from premise-free input ( 9 ; 19 ) , and controlled challenge sets such as HANS show that high benchmark accuracy can mask reliance on lexical or syntactic heuristics ( 15 ) . Similar shortcut effects appear beyond NLI, including argument reasoning artifacts ( 18 ) ; more broadly, shortcut learning is a known failure mode of modern neural systems ( 7 ) . In fact verification, 23 show analogous claim-side cues in FEVER and demonstrate that claim-only baselines remain competitive against evidence-aware models; FEVER2.0-style adversarial work further stresses robustness to perturbations ( 25 ) . A broader line of work uses contrast sets, counterfactually augmented data, and behavioral testing to expose brittle shortcut reliance ( 13 ; 6 ; 20 ) . These studies mostly examine claim-side, hypothesis-side, or local decision-boundary artifacts. NEI-CAP carries the same diagnostic stance to the evidence side and