AMREx: AMR for Explainable Fact Verification

Chathuri Jayaweera, Sangpil Youm, Bonnie Dorr · 2024

Read AMREx: AMR for Explainable Fact Verification by Chathuri Jayaweera, Sangpil Youm, Bonnie Dorr (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-01343.txt.

Description

Abstract With the advent of social media networks and the vast amount of information circulating through them, automatic fact verification is an essential component to prevent the spread of misinformation. It is even more useful to have fact verification systems that provide explanations along with their classifications to ensure accurate predictions.

Catalog record

CREATORChathuri Jayaweera, Sangpil Youm, Bonnie Dorr
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-01343

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AMREx: AMR for Explainable Fact Verification Chathuri Jayaweera, Sangpil Youm, Bonnie Dorr Abstract With the advent of social media networks and the vast amount of information circulating through them, automatic fact verification is an essential component to prevent the spread of misinformation. It is even more useful to have fact verification systems that provide explanations along with their classifications to ensure accurate predictions. To address both of these requirements, we implement AMREx, an Abstract Meaning Representation (AMR)-based veracity prediction and explanation system for fact verification using a combination of Smatch, an AMR evaluation metric to measure meaning containment and textual similarity, and demonstrate its effectiveness in producing partially explainable justifications using two community standard fact verification datasets, FEVER and AVeriTeC. AMREx surpasses the AVeriTec baseline accuracy showing the effectiveness of our approach for real-world claim verification. It follows an interpretable pipeline and returns an explainable AMR node mapping to clarify the system’s veracity predictions when applicable. We further demonstrate that AMREx output can be used to prompt LLMs to generate natural-language explanations using the AMR mappings as a guide to lessen the probability of hallucinations. 1 Introduction With the vast amount of information circulating on social media and the constantly changing Claims about various topics, automatic fact verification has become crucial for preventing the spread of misinformation. To address this need, automatic fact-checking task ( Vlachos and Riedel 2014 ) and several shared tasks have been introduced to encourage NLP researchers to develop systems that gather Evidence (Fact extraction) for a given Claim and classify it (Fact verification) as to its predicted veracity. Examples include FEVER ( Thorne et al. 2018b ; Thorne et al. 2019 ) and the current AVeriTec task Schlichtkrull et al. 2023 ; Schlichtkrull et al. 2024 , which employ the labels Supports , Refutes , NotEnoughInfo (NEI) or ConflictingEvidence/CherryPicking . Natural Language Inference (NLI) systems, which assess whether a premise semantically entails a given hypothesis ( Bowman et al. 2015 ) , have been used for fact verification, yielding demonstrably strong results in the FEVER shared task. However, there has been limited focus on the explainability of these implementations. Recent studies ( Gururangan et al. 2018 ; McCoy et al. 2019 ) have highlighted NLI models’ tendency to rely on spurious cues for entailment classification making it important to provide clear explanations alongside fact verification predictions. We design and implement a new, deterministic NLI system based on Abstract Meaning Representation (AMR), dubbed AMREx, and test it on the FEVER and AveriTeC fact-checking datasets. AMR is a rooted, directed, acyclic graph with nodes representing concepts and edges denoting the relations ( Banarescu et al. 2013 ) . This representation captures semantic relationships among entities that can be difficult to identify in a syntactic representation ( Ma et al. 2023 ) . We apply an existing AMR evaluation metric ( Cai and Knight 2013 ) , to map Claims (e.g., X was produced Y ) to relevant Evidence (e.g., X is a film produced by Y ). We incorporate this mapping into our AMREx system to yield partially explainable fact verification. We assume Evidence collection has already been completed, as our focus is on the potential for explainability of our fact-checking results, independent of the degree of correctness with respect to a ground truth. This, in fact, is the key contribution of this paper: We demonstrate that explainability is valuable regardless of performance levels. If performance is high, explainability supports an exploration of the factors contributing to the algorithm’s success. If performance is low, it serves as a diagnostic tool to understand what went wrong. Claim: Veeram was produced by Vijaya Productions. Evidence : Veeram (Valour) is a 2014 Indian Tamil action film directed by Siva and produced by Vijaya Productions. AMREx Supports AMR mapping explanation Veracity produce-01 company Vijaya Production Veeram film company name Vijaya name Siva name ARG0 name op1 op2 name op1 domain ARG1-of ARG0 op1 name ARG1-of ARG0 name op1 op2 name op1 ARG1 name produce-01 company company name name name Vijaya Vijaya film name name Veeram Veeram Evidence Claim work-of-art produce-01 produce-01 Production Production Production work-of-art direct-01 person film Veeram Figure 1: Explainable fact verification pipeline. Lower Left: AMR graph for the Claim, Lower Middle: AMR graph for the Evidence, Lower Right: The AMR graph mapping to explain the model’s prediction as “Supports” Fig. 1 illustrates our explainable output using AMR graph mapping. We quantify the degree to which the Claim AMR is contained in the Evidence AMR and present the mappings identified in this process to demonstrate whether the Claim is embedded within the Evidence. For example, the Claim Veeram was produced by Vijay Productions and Evidence Veeram (Valour) is a 2014 Indian Tamil action film directed by Siva and produced by Vijaya Productions are represented as AMRs and processed through the Smatch algorithm. This identifies similar substructures between them, showing that both texts mention a production (rooted by produce-01 predicate) with similar attributes and refer to the same film (through substructures rooted by work-of-art and film in the Claim and Evidence AMRs). AMREx uses this high-level notion of meaning containment, along with a textual similarity score, to produce the veracity prediction “Supports”. Section 2 reviews existing NLI implementations and explainable representations used in fact verification. Section 3 provides a detailed description of AMREx system and the experiments conducted. Section 4 presents an analysis and discussion of the results, with conclusions in Section 5 . 2 Related Work Below we explore existing studies related to NLI for fact verification, Explainable representation of fact verification, and AMR. 2.1 NLI for Fact Verification NLI models have been employed for fact verification by assessing whether a given premise p logically infers hypothesis h ( Bowman et al. 2015 ; Zeng and Zubiaga 2024 ) . These models usually classify Claim veracity using labels: Supports , Refutes and NEI . Thorne et al. 2018b has developed a large-scale fact verification dataset with balanced label distribution across various domains. In this study, we adopt a 3-way (FEVER) and 4-way (AVeriTec) classification for fact verification. With the development of fact verification datasets, fine-tuned language models (e.g., BERT, XLNet) have been applied to verify facts, improving generalizability without the need for manually crafted rules ( Chernyavskiy and Ilvovsky 2019 ; Nie et al. 2019 ; Portelli et al. 2020 ; Zhong et al. 2020 ) . These BERT-based models use the Claim and potential Evidence as inputs and determine the final labels. Recently, Pan et al. 2023 fine-tuned a small dataset to enhance the performance of BERT-based models, aiming to develop domain-specific models and improving generalizability. We transcend this work by employing semantic similarity in the embedding space between Claim and Evidence, along with structural similarity. Using pre-trained models, graph neural networks (GNNs) have been employed to enhance reasoning for fact verification ( Zhong et al. 2020 ; Zhou et al. 2019 ) . These models represent Evidence as nodes within a graph, enabling information exchange between nodes, thereby improving reasoning capabilities to determine the final label. Zhong et al. 2020 use Semantic Role Labeling (SRL), assigning semantic roles to both Claim and Evidence sentences for graph construction. Building on the concept of deeper reasoning for fact verification,

Read the complete text