FinGrAct: A Framework for FINe-GRrained Evaluation of ACTionability in Explainable Automatic Fact-Checking
Islam Eldifrawi, Shengrui Wang, Amine Trabelsi · 2025
Read FinGrAct: A Framework for FINe-GRrained Evaluation of ACTionability in Explainable Automatic Fact-Checking by Islam Eldifrawi, Shengrui Wang, Amine Trabelsi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05229.txt.
Introduction
Islam Eldifrawi, Shengrui Wang and Amine Trabelsi wrote a 2025 paper on how to judge fact-checking systems. Such systems now add short notes that explain their verdicts. A good note should help a reader act, not just sound clear. The authors say no prior method scored that trait on its own.
Their tool, FinGrAct, can search the web while it scores a note. It uses set rules and a test set built for this task. In trials, its scores matched human judges more closely than rival tools did. It also showed less bias toward its own style. The tool gives builders a firmer way to test their notes.
Catalog record
| CREATOR | Islam Eldifrawi, Shengrui Wang, Amine Trabelsi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-05229
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FinGrAct: A Framework for FINe-GRrained Evaluation of ACTionability in Explainable Automatic Fact-Checking Islam Eldifrawi, Shengrui Wang, Amine Trabelsi Abstract The field of explainable Automatic Fact-Checking (AFC) aims to enhance the transparency and trustworthiness of automated fact-verification systems by providing clear and comprehensible explanations. However, the effectiveness of these explanations depends on their actionability—their ability to empower users to make informed decisions and mitigate misinformation. Despite actionability being a critical property of high-quality explanations, no prior research has proposed a dedicated method to evaluate it. This paper introduces FinGrAct, a fine-grained evaluation framework that can access the web and it is designed to assess actionability in AFC explanations through well-defined criteria and an evaluation dataset. FinGrAct surpasses state-of-the-art (SOTA) evaluators, achieving the highest Pearson and Kendall correlation with human judgments while demonstrating the lowest ego-centric bias, making it a more robust evaluation approach for actionability evaluation in AFC. 1 Introduction Explanation of claim veracity is essential in automated fact-checking as it enhances transparency, fosters trust, and educates users by clarifying why a claim is deemed true or not. It provides contextual understanding, and prevents the spread of misinformation. In the domain of explainable automated fact-checking (AFC), the explanation of a claim’s predicted veracity is assessed based on the presence of specific desired properties, referred to as "desiderata," as outlined by Kotonya and Toni (2020a) . While some of these desiderata have been explored in the field of summarization, others, such as actionability, a critical property in fact-checking, remain unexplored. To date, no automatic evaluator for actionability has been developed. However, some general purpose SOTA evaluators emerged, but only for summarization tasks. In this work, these evaluators are adapted to AFC. Figure 1: FinGrAct Flow . Input : Claim, Evidence, Label, Explanation to be evaluated (with varying degrees of actionability or none). Processing : FinGrAct Error Segmentation and Correction, FinGrAct Evaluator, FinGrAct Source Evaluator, URL Content Retriever (UCR). Output : 3 evaluation scores (Error Detection, Error Correction, Supporting Links Degrees) classified as None (0), Partial (1), Full (2), aggregated into a Likert scale score (1 to 5). Actionability remains underexplored due to its complexity. According to Kotonya and Toni (2020a) , actionability in AFC refers to providing factual corrections for identified errors in a non-factual claim, supported by reliable sources and references . This suggests that actionability is highly correlated with other key properties, such as relevance to the claim and completeness, where the explanation must provide a comprehensive context for why the claim is considered true or not. Evaluating actionability presents significant challenges as it requires well-defined criteria for judgment, as well as access to external sources in the internet to verify the reliability of supporting references. Figure 1 shows examples of explanations with different degrees of actionability. To tackle the challenge of automatically evaluating the actionability of explanations in AFC systems, we introduce FinGrAct. FinGrAct is a fine-grained auto-evaluator that systematically measures the degree of actionability in these explanations. The evaluation framework is illustrated in Figure 1 . Our contributions are as follows: 1. We present a dataset, constructed from existing sources, containing explanations for claim labels with varying degrees of actionability. Additionally, it includes human-rated actionability scores, which can serve as benchmarks for evaluating the performance of different evaluators of actionability. 2. We present FinGrAct a novel fine-grained evaluator of explanations for actionability within the context of explainable AFC, based on LLM prompting, that correlates better with human ratings than other adapted SOTA evaluators. In addition, we showed that adding a simple component, named the URL Content Retriever (UCR), to fetch and evaluate web-link textual content, enhanced the performance of actionability evaluation in all SOTA evaluators. UCR enables LLMs to assess the validity and relevance of the supporting sources referenced in explanations. 3. Furthermore, we conduct an investigation into the ego-centric bias present in LLMs. ego-centric bias happens when LLMs acting as judges to the output of various LLMs including themselves, tend to assign higher scores to their own output. In addition, FinGrAct has the least ego-centric bias among all other adapted SOTA evaluators. 2 Related Work With the emergence of LLMs such as ChatGPT, recent studies have employed these models as evaluators Fu et al. (2024) to assess LLM performance across various benchmarks, including G-EVAL Liu et al. (2023) , and FineSurE Song et al. (2024a) . Nonetheless, it has been observed that LLMs functioning as auto-raters tend to show a preference for their own generated responses Panickssery et al. (2024) . This is called ego-centric bias. Training open-source general-purpose LLM autoraters has been investigated recently. TIGERScore, a Llama-2 model trained on GPT-4 generated error analysis data across multiple tasks, such as summarization, translation, and instruction-following, is presented by Jiang et al. 2023 . However, it doesn’t follow a likert-scale scoring or can it be scaled to it. For instance, it can produce a score of -12 making it hard to be compared to other evaluators. Prometheus Kim et al. (2023) , InstructScore Xu et al. (2023) , and Prometheus-2 Kim et al. (2024) are comparable methods. Vu et al. 2024 developed reward models used for aligning LLMs to human preferences via reinforcement learning and called their model FLAME. However, they haven’t published their model or their dataset collection. SOTA evaluators like G-Eval and Prometheus are widely used in summarization to assess properties such as coherence and faithfulness. Unlike property-specific tools like FineSurE, they are adaptable for evaluating new attributes. FineSurE gives a specific fine-grained definition for each property it measures in a summary based on the key-facts it found in the transcript. Making its adaption to new tasks and new properties very hard. All the mentioned work is directed towards evaluating certain desired properties in diverse tasks like summarization and question answering. However, no effort has been directed to date towards developing an autorater for the desiderata of the explanations of AFC except when Feher et al. (2025) tried to evaluate properties like (self)-contradiction, hallucination, convincingness and overall quality. However, they didn‘t address the properties mentioned by Kotonya and Toni (2020a) ; Eldifrawi et al. (2024) that were deemed critical for fact-checking explanation like actionability for instance. 3 FinGrAct Framework 3.1 Implementation Details Inspired by recent advancements in fine-grained evaluation criteria for summarization—particularly in assessing key properties such as faithfulness and completeness, as demonstrated in works like Ye et al. (2023) ; Zhang et al. (2024) ; Song et al. (2024a) —we propose a specialized fine-grained evaluation methodology designed to assess the actionability of AFC explanations. Our approach, similar to these studies, leverages targeted prompting of specific LLMs to ensure a rigorous and nuanced assessment. As depicted in Figure 1 , we propose a divide-and-conquer approach to evaluating the actionability of an explanation—given a false claim and supporting evidence—by breaking it down into three distinct tasks: Error Segmentation and Correction , Explanation Evaluation , and Source Evaluation . We have the detailed prompts for every task in the Appendix E . Error Segmentation and Correction: This task is necessary for FinGrAct to preprocess a claim, extract what it identifies as factual errors , and determine how corrections within this claim should be made based on reliable given evidence (see Figure 1 for an example). Once this information