VeriFastScore : Speeding up long-form factuality evaluation

Rishanth Rajendhran, Amir Zadeh, Matthew Sarte, Chuan Li, Mohit Iyyer · 2025

Read VeriFastScore : Speeding up long-form factuality evaluation by Rishanth Rajendhran, Amir Zadeh, Matthew Sarte, Chuan Li, Mohit Iyyer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16973.txt.

Introduction

Rishanth Rajendhran, Amir Zadeh, Matthew Sarte, Chuan Li, and Mohit Iyyer speed up long answer fact checks, in work from 2025. Popular fact scores split a long answer into small claims and check each claim alone. That step by step route can take over a hundred seconds for one answer, which blocks large scale use.

Their model learns to pull and check all claims in one pass from search proof. Fake training data tuned a mid size open model for the joint task. Few shot prompts with closed models failed on the long proof load. The tuned model matched the slow pipeline closely at answer and system levels. Speed rose more than six fold in the full test runs.

Catalog record

CREATORRishanth Rajendhran, Amir Zadeh, Matthew Sarte, Chuan Li, Mohit Iyyer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-16973

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VeriFastScore : Speeding up long-form factuality evaluation Rishanth Rajendhran, Amir Zadeh, Matthew Sarte, Chuan Li, Mohit Iyyer Abstract Metrics like FactScore and VeriScore that evaluate long-form factuality operate by decomposing an input response into atomic claims and then individually verifying each claim. While effective and interpretable, these methods incur numerous LLM calls and can take upwards of 100 seconds to evaluate a single response, limiting their practicality in large-scale evaluation and training scenarios. To address this, we propose VeriFastScore , which leverages synthetic data to fine-tune Llama3.1 8B for simultaneously extracting and verifying all verifiable claims within a given text based on evidence from Google Search. We show that this task cannot be solved via few-shot prompting with closed LLMs due to its complexity: the model receives ∼ \sim 4K tokens of evidence on average and needs to concurrently decompose claims, judge their verifiability, and verify them against noisy evidence. However, our fine-tuned VeriFastScore model demonstrates strong correlation with the original VeriScore pipeline at both the example level ( r = 0.80 r=\mathbf{0.80} ) and system level ( r = 0.94 r=\mathbf{0.94} ) while achieving an overall speedup of 6.6 × \times (9.9 × \times excluding evidence retrieval) over VeriScore . To facilitate future factuality research, we publicly release our VeriFastScore model and synthetic datasets. 1 1 1 Code and data available at https://github.com/RishanthRajendhran/VeriFastScore . 1 Introduction Modern methods to evaluate the factuality of long-form generation, such as FactScore Min et al. 2023 , Safe Wei et al. 2024 , and VeriScore Song et al. 2024 , rely on a pipelined approach that uses LLMs to decompose an input text into “atomic” claims and then verify each claim against retrieved evidence documents. While this pipeline results in interpretable outputs that correlate strongly with human annotations of factuality, it is also slow Liu et al. 2025 , requiring many calls to large models to evaluate a text, which limits its usability in both evaluation and training (e.g., as a reward model in RLHF). In this paper, we propose a significantly faster alternative: a single-pass factuality evaluator that simultaneously performs claim decomposition and verification over a long-form input. Our method, VeriFastScore , is trained to jointly extract all verifiable claims from a response and assess their factuality against retrieved evidence. To train VeriFastScore , we collate sentence-level evidence with claim-level outputs produced by VeriScore , a high-quality general-domain factuality evaluation metric, into a synthetic dataset used to fine-tune Llama3.1 8B Instruct Grattafiori et al. 2024 . Each training example contains a response to be evaluated as well as a corresponding evidence context that consists of web search results retrieved by querying individual sentences in the response. VeriFastScore is then asked to output a list of verifiable claims from the response, each labeled as Supported or Unsupported based on the evidence (see Figure 1 ). This task is non-trivial: the model must internally decontextualize the response, identify atomic claims, resolve coreferences, and assess support against an evidence context spanning up to 4K tokens of potentially noisy information. Standard few-shot prompting of powerful closed models like GPT-4o struggles with this complexity, achieving a Pearson correlation of only 0.33 with VeriScore despite high API costs. Figure 1: (left) The VeriScore pipeline Song et al. 2024 involves a separate call to a trained model (or LLM API) to decompose every sentence of the long-form response. More model calls are issued for each decomposed claim to verify them against claim-level evidence retrieved from Google Search. In contrast, our VeriFastScore method retrieves sentence-level evidence from Google Search, combines it together with the response into a single long-context prompt, and outputs a list of claims and labels with just a single call to a trained model. Despite these challenges, VeriFastScore matches the quality of VeriScore closely while offering substantial improvements in both cost and runtime efficiency: it achieves a Pearson correlation of \mathbf{0.80} (Pearson r, p<0.001) with VeriScore ’s scores, a 9.9 \times modeling speedup and a 6.64 \times overall speedup in wall-clock time, reducing both modeling and retrieval latency while maintaining the metric’s interpretability. Human evaluation corroborates the high quality of outputs from VeriFastScore . We release VeriFastScore and our synthetic training dataset to support efficient, interpretable factuality evaluation and facilitate its use in alignment settings. 2 Building a faster factuality metric In this section, we first describe the limitations of existing factuality metrics like VeriScore , whose separate decomposition, retrieval, and verification stages require several calls to LLMs to complete for a single response. These limitations motivate our design and development of VeriFastScore , which combines all steps in the existing factuality evaluation pipeline into a single model call. To enable such a fast evaluation, we generate synthetic data from VeriScore and use it to fine-tune a language model on response-level decomposition and verification. Prompt Source Split # Responses Avg. Claims/Response Avg. Token Length Claim Verification Label (%) Response Evidence Facts Supported Unsupported VeriScore data Train 4082 16 298 4381 271 68 32 Test 1815 16 301 4466 278 68 32 Tulu3 Personas Train 4860 20 366 3755 334 59 41 Test 4083 20 370 3714 329 59 41 Table 1: Overview of the datasets used. For each data source and split (train/test), we report the total number of model responses, the average number of verifiable claims per response, average token lengths 3 3 3 Token lengths are estimated using Llama 3.1 Instruct tokenizer. of model responses, consolidated evidence, and consolidated list of verifiable claims extracted by VeriScore from model responses, and the percentage of claims labeled by VeriScore as supported and unsupported by retrieved evidence. Labels are more balanced in the dataset collected during this work using prompts from Tulu3 Personas dataset. 2.1 The “decompose and verify” pipeline for factuality evaluation Recently-proposed metrics for long-form factuality evaluation such as FactScore and Safe are implemented by the following multi-stage pipeline: 1. Claim extraction: decompose the long-form model response into “atomic” (i.e., short) claims 2. Evidence retrieval: collect evidence for every claim individually by retrieving relevant documents from a datastore or search engine 3. Claim verification: mark each atomic claim as either supported or unsupported by the evidence collected for that claim In FactScore and VeriScore , the first and third stages are implemented by either open-weight or closed LLMs (e.g., GPT-4), while SAFE’s decomposition and verification stages are implemented by prompting closed models from OpenAI or Claude. VeriScore , our starting point: Unlike FactScore Min et al. 2023 and Safe Wei et al. 2024 , which attempt to verify all atomic claims in a model’s response using external evidence, the decomposition stage of VeriScore focuses solely on verifiable claims—defined as Verifiable claims describe a single event or state 4 4 4 Event: change of state, for example, “Jensen Huang founded NVIDIA in 1993 in California, U.S.” State: for example, “Westborough is a town in Worcester, MA.” with all necessary modifiers (e.g., spatial, temporal, or relative clauses) that help denote entities or events in the real world. They should be verifiable against reliable external sources (e.g., Wikipedia). They should exclude personal experiences, subjective opinions, hypotheticals, suggestions, advice, or instructions. ( Maienborn 2003 ; Maienborn 2019 ) Like Safe , VeriScore also relies on Google Search to retrieve evidence

Read the complete text