VISTA Score: Verification In Sequential Turn-based Assessment

Ashley Lewis, Andrew Perrault, Eric Fosler-Lussier, Michael White · 2025

Read VISTA Score: Verification In Sequential Turn-based Assessment by Ashley Lewis, Andrew Perrault, Eric Fosler-Lussier, Michael White (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-27052.txt.

Introduction

Ashley Lewis, Andrew Perrault, Eric Fosler-Lussier, and Michael White introduce VISTA, a framework for checking factual accuracy across a whole conversation. Hallucination, meaning statements with no support in evidence or context, blocks chat systems from trusted use. Older scores judge single replies in isolation or treat anything unverifiable as simply wrong, missing how errors build over turns.

VISTA splits each assistant reply into single factual claims and checks each against trusted sources and the chat history. Unverifiable claims are sorted into kinds such as subjective, contradicted, or lacking evidence. Across eight models and four benchmarks it detects hallucination better than prior methods. Human checks confirm clearer judgments, making conversation truthfulness easier to track and compare.

Catalog record

CREATORAshley Lewis, Andrew Perrault, Eric Fosler-Lussier, Michael White
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-27052

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

VISTA Score: Verification In Sequential Turn-based Assessment Ashley Lewis, Andrew Perrault, Eric Fosler-Lussier, Michael White Abstract Hallucination—defined here as generating statements unsupported or contradicted by available evidence or conversational context—remains a major obstacle to deploying conversational AI systems in settings that demand factual reliability. Existing metrics either evaluate isolated responses or treat unverifiable content as errors, limiting their use for multi-turn dialogue. We introduce VISTA (Verification In Sequential Turn-based Assessment), a framework for evaluating conversational factuality through claim-level verification and sequential consistency tracking. VISTA decomposes each assistant turn into atomic factual claims, verifies them against trusted sources and dialogue history, and categorizes unverifiable statements (subjective, contradicted, lacking evidence, or abstaining). Across eight large language models and four dialogue factuality benchmarks ( Ais , Begin , FaithDial , and Fade ), Vista substantially improves hallucination detection over FactScore and LLM-as-Judge baselines. Human evaluation confirms that VISTA’s decomposition improves annotator agreement and reveals inconsistencies in existing benchmarks. By modeling factuality as a dynamic property of conversation, VISTA offers a more transparent, human-aligned measure of truthfulness in dialogue systems. 1 Introduction Figure 1: Overview of the VISTA Score pipeline. Each assistant turn undergoes claim extraction, verification, and categorization, with accumulated background facts informing subsequent turns. Despite advances in reasoning and retrieval, large language models (LLMs) still hallucinate, producing fluent but false statements that erode user trust and limit deployment in factual settings. Existing metrics often treat each generation as isolated text, ignoring the sequential and pragmatic nature of dialogue, where earlier claims constrain or ground later ones. Recent analyses Kalai et al. 2025 further show that factuality progress depends on distinguishing abstentions , or when a model declines to answer, from hallucinations , which assert falsehoods with confidence. We argue that factuality in dialogue should be modeled as a dynamic, turn-based process rather than a static property of text. To do so, we propose VISTA Score ( Verification In Sequential Turn-based Assessment ), a framework that reconceptualizes hallucination detection as sequential claim verification. Each assistant turn is decomposed into atomic factual statements, verified against trusted references and dialogue history, and categorized by unverifiability type: contradicted , lacking evidence , subjective , or abstaining . This decomposition reveals interactions between factual precision, dialogue consistency, and pragmatic appropriateness that prior metrics collapse. Unlike prior work (FActScore Min et al. 2023 , LLM-as-Judge Zha et al. 2023 ; Peisakhovsky et al. 2025 ), which evaluate isolated responses or conflate unverifiable and subjective content, VISTA captures factual reliability as it unfolds across turns. While it also leverages LLMs for claim extraction and verification, these models operate within a structured, turn-based pipeline rather than as monolithic judges, bridging claim-level verification and conversational coherence. From a linguistic perspective, this approach aligns with the notion of common ground , or the evolving set of propositions treated as shared knowledge Stalnaker 1978 ; Clark 1996 , but adopts a stricter formulation: VISTA tracks what can be established as verifiably true, not merely believed. Verified and acceptable claims expand this verifiable discourse state, while contradicted or unsupported claims signal breakdowns in factual grounding. We evaluate VISTA on four retrieval-augmented dialogue benchmarks (AIS, BEGIN, FaithDial, FADE) Rashkin et al. 2023 ; Dziri et al. 2022b ; Das et al. 2022 ; Dziri et al. 2022a and eight open- and closed-weight LLMs, including GPT, Llama, Deepseek, Qwen, and Mistral. VISTA achieves higher hallucination-detection accuracy than existing metrics, with exceptionally large boosts for smaller, open-weight models. Human evaluation confirms that claim-level decomposition improves annotator agreement and exposes inconsistencies in current benchmarks. Our contributions are: • A dialogue-aware factuality framework that integrates claim verification with sequential consistency tracking, providing fine-grained, interpretable labels that distinguish factual errors, abstentions, and opinions. • Empirical evidence that VISTA improves factuality assessment across diverse models and datasets. • A human evaluation showing that claim-level decomposition increases annotation reliability and clarifies benchmark inconsistencies. • A dataset of 140 annotated conversations supporting future research on conversational factuality. By reframing factuality as an evolving property of conversation, VISTA moves beyond static correctness toward a more transparent and human-aligned measure of truthfulness in dialogue systems. Our implementation and dataset are publicly available at [GITHUB LINK] . 2 Related Work Decomposition-Based Methods. Recent approaches to hallucination detection emphasize breaking model outputs into smaller factual units for verification. This decomposition allows evaluators to isolate true and false claims that may coexist in a single generation. Min et al. 2023 introduced FActScore , which decomposes text into atomic factual statements and verifies each against a trusted source to produce a fine-grained score. Zha et al. 2023 extended this idea through AlignScore , a learned alignment model that assesses factual consistency across diverse tasks such as NLI and QA. While both enable interpretable, claim-level judgments, they treat all unverifiable, abstaining, or subjective statements as hallucinations—a simplification poorly suited to dialogue, which often blends factual, uncertain, and opinionated content. Within dialogue, Chen et al. 2025 proposed FineDialFact , a benchmark for fine-grained fact verification that explicitly labels subjective or insufficiently supported claims as “not enough information.” However, this conflates subjectivity with evidential uncertainty, limiting diagnostic value. VISTA builds on this decomposition tradition but introduces a pragmatic distinction between unverifiable subjective claims and unsupported factual ones. By modeling these distinctions sequentially across dialogue turns, it provides a more faithful view of factual reliability in conversation. LLM-as-Judge Approaches. Another prominent line of work uses LLMs themselves as factuality evaluators. Liu et al. 2023 and others demonstrated that models such as GPT-4 can produce reliable judgments of coherence and factual accuracy. In hallucination detection, Manakul et al. 2023 introduced SelfCheckGPT , which compares a model’s output with resampled continuations to flag unsupported content. More recently, Peisakhovsky et al. 2025 proposed the FINAL benchmark, which reframes hallucination detection as localizing factual inconsistencies through natural-language explanations. Despite their scalability, these LLM-as-judge methods remain sensitive to prompt design, model bias, and domain familiarity—limitations that also affect decomposition-based approaches. VISTA embeds LLM evaluators within a structured, multi-stage pipeline, narrowing the model’s role to subtasks of claim extraction, verification, and categorization. Rather than relying on judgments from a single model pass, this decomposition facilitates clearer analysis and more consistent evaluation, and can be implemented with open-weight models for greater accessibility and reproducibility. Benchmarks for Dialogue Factuality. Several benchmarks have advanced the study of factuality in dialogue. Dziri et al. 2022b introduced BEGIN , augmenting Wizard-of-Wikipedia with hallucination labels. The AIS benchmark Rashkin et al. 2023 evaluates attribution to identified sources, while FaithDial Dziri et al. 2022a provides human-corrected faithful dialogue variants. FADE Das et al. 2022 synthetically introduces factual errors, and DialFact Gupta et al. 2022 formulates dialogue verification as fact-checking against Wikipedia. These datasets standardize factuality assessment but largely operate at the single-turn level, overlooking how factual claims evolve or contradict each other throughout a conversation. VISTA is designed specifically for this sequential setting: it tracks verified information across turns, detects contradictions, and distinguishes genuine hallucinations from subjectivity or abstention. Other Methods. Beyond decomposition and LLM-judging, other approaches estimate or predict hallucination risk through auxiliary signals. Uncertainty-based methods such as semantic entropy Farquhar et al. 2024 gauge model confidence by measuring semantic variability across generations, while entailment-based techniques like FactCC Kryscinski et al. 2020 and Q 2

Read the complete text