When Verification Fails: How Compositionally Infeasible Claims Escape Rejection
Muxin LiuDelip RaoGrace KimChris Callison-BurchUniversity of Pennsylvania [email protected] · 2026
Read When Verification Fails: How Compositionally Infeasible Claims Escape Rejection by Muxin LiuDelip RaoGrace KimChris Callison-BurchUniversity of Pennsylvania [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10990.txt.
Introduction
Liu and three colleagues show that current science-claim checking benchmarks cannot separate strict models from shortcut models. Strict models enforce the closed-world rule, accepting a claim only when every constraint is fully backed. Shortcut models test only the most salient constraint. Current benchmarks build false claims by changing one salient element, so they cannot tell the two apart. The authors build combined false claims with a supported salient constraint and a contradicted non-salient one. Models that saturate current benchmarks still over-accept these claims.
Catalog record
| CREATOR | Muxin LiuDelip RaoGrace KimChris Callison-BurchUniversity of Pennsylvania [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60464 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10990
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When Verification Fails: How Compositionally Infeasible Claims Escape Rejection Muxin LiuDelip RaoGrace KimChris Callison-BurchUniversity of Pennsylvania [email protected] Abstract Scientific claim verification, the task of determining whether claims are entailed by scientific evidence, is fundamental to establishing discoveries in evidence while preventing misinformation. This process involves evaluating each asserted constraint against validated evidence. Under the Closed-World Assumption (CWA), a claim is accepted if and only if all asserted constraints are positively supported. We show that existing verification benchmarks cannot distinguish models enforcing this standard from models applying a simpler shortcut called salient-constraint checking, which applies CWA’s rejection criterion only to the most salient constraint and accepts when that constraint is supported. Because existing benchmarks construct infeasible claims by perturbing a single salient element they are insufficient at distinguishing between rigorous claim verification and simple salient-constraint reliance. To separate the two, we construct compositionally infeasible claims where the salient constraint is supported but a non-salient constraint is contradicted. Across model families and modalities, models that otherwise saturate existing benchmarks consistently over-accept these claims, confirming the prevalence of such shortcut reasoning. Via model context interventions, we show that different models and prompting strategies occupy distinct positions on a shared ROC curve, indicating that the gap between model families reflects differences in verification threshold rather than underlying reasoning ability, and that the compositional inference bottleneck is a structural property of current verification behavior that strategy guidance alone cannot overcome. 1 Introduction Claim verification, the task of determining whether claims are entailed by evidence, is fundamental to grounding discoveries in evidence while preventing misinformation. Claim verification has been widely studied as a natural language inference (NLI) task, with large-scale benchmarks spanning general knowledge verification ( 49 ; 4 ) , scientific literature ( 55 ; 56 ) , clinical trials ( 23 ; 54 ) , tables ( 9 ; 32 ) , and charts ( 2 ; 3 ; 34 ; 60 ) . This judgment requires a precise epistemic standard: a claim should be accepted if and only if the evidence provides positive support for it, and absence of support warrants rejection. This standard is formalized as the Closed-World Assumption (CWA) ( 12 ; 41 ) . Operationally, this requires checking every constraint the claim asserts ( 37 ) , including those derived compositionally from multiple evidence units ( 46 ; 64 ; 8 ) . We evaluate Large Language Models (LLMs) across three established benchmarks, NLI4CT ( 23 ; 22 ) for clinical trial text, SCITAB ( 32 ) for scientific tables, and SciVer ( 60 ) for charts, and we find that models consistently achieve high accuracy on infeasible claims but lower accuracy on feasible ones, as shown in Figure 2 . This asymmetry could be interpreted as evidence of conservative, closed-world verification, or could reflect a weaker shortcut observed in the literature for other tasks as well ( 18 ; 35 ; 31 ) . The construction of existing benchmarks makes full CWA and a simpler shortcut procedure behaviorally indistinguishable. Negative examples in these benchmarks (also called standard negatives) are created by perturbing a single salient element, such as flipping a number, reversing a trend, or swapping an entity label. This makes the perturbed constraint both the most checkable part of the claim and sufficient for rejection. As a result, as shown in Figure 1 , a shortcut is sufficient when verifying Existing Benchmark Infeasible Claim : identify the most salient constraint in the claim, check whether it is supported by the evidence, and decide on that basis alone. This salient-constraint checking procedure is an easier, less formal approximation of CWA, as it applies CWA’s rejection criterion but only to the most prominent constraint, skipping the exhaustive check that full CWA requires. It also explains the observed asymmetry. For infeasible claims, the salient constraint is violated, so rejection follows directly. For feasible claims, acceptance requires the salient constraint to be clearly retrievable, and any uncertainty defaults to rejection instead. High negative accuracy together with lower positive accuracy is therefore consistent with both full CWA and salient-constraint checking. Figure 1: CWA verification checks all constraints; salient-constraint checking only finds the most prominent constraint. On existing benchmarks both agree since violated constraints are always salient. On compositionally infeasible claims, salient-constraint checking accepts while CWA rejects. To expose the gap between salient-constraint checking and full CWA, we construct claims for which the shortcut yields acceptance but CWA requires rejection, as shown in Figure 1 , under the Compositionally Infeasible Claim setting. These claims, which we call compositionally infeasible claims (also called adversarial negatives), are designed so that the salient constraint appears supported by the evidence, while the violation lies in a non-salient constraint such as a composed relationship across multiple parts of the evidence. We create hard negatives using a graph-based corrupt-and-propagate procedure, which preserves the salient observations while introducing violations through composing multiple parts of the same evidence source such as two sentences in a clinical report, two trends in a chart, or two columns in a table. Salient-constraint checking verifies a single constraint and accepts, while full CWA checks all constraints including non-salient ones and rejects. Figure 2: Averaged accuracy across models. Compositionally feasible claims score higher ( \uparrow ) than standard positives; compositionally infeasible claims score lower ( \downarrow ). Compositional structure shifts behavior toward acceptance. Models that perform strongly on existing benchmarks fail consistently on adversarial examples, as summarized in Figure 2 : they accept adversarial negatives at higher rates than standard negatives, and this gap persists across model families and scales. Models accept claims when the salient constraint appears supported, which produces systematic over-acceptance on adversarial negatives. We further study this failure with test-time guidance toward CWA, including closed-world instructions and explicit decomposition. These interventions shift the tradeoff between acceptance and rejection rather than improving performance on adversarial negatives overall. This movement along the ROC curve suggests that different guidance strategies change how the model applies verification, but do not resolve the underlying reasoning bottleneck. To analyze reasoning more directly, we also introduce a rubric-based LLM judge that scores how well a model’s reasoning trace follows CWA, including whether it checks all constraints the claim asserts or terminates early at the salient one. These scores reveal systematic differences across models, examples, and interventions, and provide a trace-level analysis of verification reasoning beyond outcome-based evaluation. We make three contributions. First, we identify a structural limitation of existing verification benchmarks: single-constraint construction makes salient-constraint checking and full CWA behaviorally indistinguishable. Second, we construct compositionally infeasible claims across three modalities that expose this gap, and show consistent over-acceptance across model families and scales. Third, we show this failure reflects a reasoning bottleneck rather than a strategy choice through both intervention and trace analysis. 2 Theoretical Framework We formalize the verification standards for scientific claim verification, introduce salient-constraint checking as a weaker procedure that existing benchmarks cannot distinguish from full CWA, and define compositional falsification as the condition that separates the two 2.1 Closed-World and Open-World Assumptions Let E=\{e_{1},\ldots,e_{m}\} be a set of evidence pieces and c a claim asserting constraints \mathcal{C}(c)=\{c_{1},\ldots,c_{n}\} . Under CWA , c is accepted iff every constraint is supported: \text{CWA}(c,E)=\begin{cases}\text{Accept}&\text{if }\forall c_{i}\in\mathcal{C}(c),\ \exists S_{i}\subseteq E\ \text{s.t.}\ \bigwedge_{e\in