Is Conformal Factuality for RAG-based LLMs Robust? Novel Metrics and Systematic Insights

Yi Chen, Daiwei Chen, Sukrut Madhav Chikodikar, Caitlyn Heqi Yin, Ramya Korlakai Vinayak · 2026

Read Is Conformal Factuality for RAG-based LLMs Robust? Novel Metrics and Systematic Insights by Yi Chen, Daiwei Chen, Sukrut Madhav Chikodikar, Caitlyn Heqi Yin, Ramya Korlakai Vinayak (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16817.txt.

Introduction

Chen and co-authors test whether conformal factuality stays reliable for retrieval-augmented language models. Such systems ground answers in retrieved text, then filter single claims with a calibrated threshold. The authors study this filtering across generation, scoring, calibration, robustness, and cost. They propose new metrics that also weigh how informative the output stays. Across three benchmarks, they find filtering often yields empty, useless output at high factuality. They also find the guarantee breaks under distribution shift and distractors. Light entailment verifiers matched costly model scorers at far less compute.

Catalog record

CREATORYi Chen, Daiwei Chen, Sukrut Madhav Chikodikar, Caitlyn Heqi Yin, Ramya Korlakai Vinayak
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE107700
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.16817

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Is Conformal Factuality for RAG-based LLMs Robust? Novel Metrics and Systematic Insights Yi Chen, Daiwei Chen, Sukrut Madhav Chikodikar, Caitlyn Heqi Yin, Ramya Korlakai Vinayak Abstract Large language models (LLMs) frequently hallucinate, limiting their reliability in knowledge-intensive applications. Retrieval-augmented generation (RAG) and conformal factuality have emerged as potential ways to address this limitation. While RAG aims to ground responses in retrieved evidence, it provides no statistical guarantee that the final output is correct. Conformal factuality filtering offers distribution-free statistical reliability by scoring and filtering atomic claims using a threshold calibrated on held-out data, however, the informativeness of the final output is not guaranteed. We systematically analyze the reliability and usefulness of conformal factuality for RAG-based LLMs across generation, scoring, calibration, robustness, and efficiency. We propose novel informativeness-aware metrics that better reflect task utility under conformal filtering. Across three benchmarks and multiple model families, we find that (i) conformal filtering suffers from low usefulness at high factuality levels due to vacuous outputs, (ii) conformal factuality guarantee is not robust to distribution shifts and distractors , highlighting the limitation that requires calibration data to closely match deployment conditions, and (iii) lightweight entailment-based verifiers match or outperform LLM-based model confidence scorers while requiring over 100 × 100\times fewer FLOPs. Overall, our results expose factuality–informativeness trade-offs and fragility of conformal filtering framework under distribution shifts and distractors , highlighting the need for new approaches for reliability with robustness and usefulness as key metrics, and provide actionable guidance for building RAG pipelines that are both reliable and computationally efficient. Keywords RAG \cdot LLM \cdot hallucination mitigation \cdot conformal prediction \cdot factuality guarantee \cdot calibration 1 Introduction Large language models (LLMs) have demonstrated remarkable capabilities across open-domain question answering, reasoning, and scientific discovery ( 6 ; 13 ; 39 ) . Yet, a persistent barrier to their reliable deployment is the phenomenon of hallucinations : outputs that are fluent and confident but factually incorrect ( 17 ; 27 ; 16 ) . Such errors are not merely cosmetic. In safety-critical settings, such as medicine, law, or finance, a single fabricated claim can erode trust, propagate misinformation, and incur high societal or financial costs. This makes hallucination mitigation one of the central challenges in advancing trustworthy LLMs. A rich body of work has emerged to address this challenge along two main directions: (1) retrieval-augmented generation (RAG) and (2) conformal methods. RAG aims to reduce hallucinations by grounding responses in trusted external knowledge sources, typically by conditioning generation on retrieved passages ( 21 ; 12 ; 32 ) . While RAG reduces the likelihood of unsupported claims, it does not offer statistical guarantees on the factuality of the final response. Even with reference, LLMs can produce hallucinations in the generated response ( 16 ) . The conformal prediction (CP) framework, on the other hand, aims to provide a statistical guarantee on the final output, often via post-processing of the initial LLM response. CP frameworks usually first decompose the initial LLM output into atomic claims, score each claim with a factuality scoring function, and filter those falling below a threshold determined using a calibration dataset ( 26 ; 8 ) . This procedure provides formal coverage guarantees but often at the expense of informativeness, since aggressive filtering may yield empty or vacuous outputs. Furthermore, CP filtering cannot improve the usefulness or accuracy of the LLM response; it can only remove hallucinations. Despite their complementary strengths, it is unclear whether conformal prediction (CP) can improve reliability of RAG-based LLMs. While several recent works integrate RAG with CP methods ( 22 ; 30 ; 11 ) , they fall short of a systematic analysis that disentangles where gains come from and when guarantees break down. A comprehensive understanding of the strengths and limitations of this combination requires not only integrating the two frameworks, but also developing principled evaluation. In particular, standard metrics such as empirical factuality can exhibit multiple failure modes and obscure real utility—for example, an empty answer is trivially “factually correct” under such measures. Additionally, it remains unclear whether improved factuality necessarily requires larger, more computationally expensive verifiers, or whether lightweight alternatives can achieve comparable or superior performance. This raises a scaling-law-style question for factuality filtering: how do reliability and utility scale with verifier capacity and inference cost, and where are the diminishing returns? Understanding the relationship between factuality, model scale, and computational cost is critical for deploying reliable LLM systems in practice, where both latency and compute budgets constrain end-to-end RAG pipelines. We address these issues in this work. Our Contributions : We systematically investigate the conformal factuality filtering framework for RAG-based LLMs (Figure 1 ) making the following contributions: • We propose novel metrics to capture informativeness component that is often missed by traditional metrics: non-empty rate and non-vacuous empirical factuality , which jointly capture both correctness and information retention, and sufficient correctness and conditional sufficient correctness , which measure whether an output contains enough correct information to infer the final answer to the query. These novel metrics capture the trade-off between factual correctness and informativeness, providing practical insights and tools for future work on hallucination mitigation. • We conduct comprehensive evaluation that spans diverse datasets (questions with free form answers, math and natural question answering), various open-source model families and sizes (with and without reasoning), and scoring functions (entailment-based and LLM-based scorers). Furthermore, we evaluate robustness against distribution shifts and distractors, shedding light on both the capabilities and limitations of this approach. Our evaluation highlights the key issues – limited usefulness at high factuality levels and non-robustness to distribution shifts and distractors . This raises the need for new approaches for guaranteeing factuality with usefulness and robustness as important metrics. • In addition, we analyze the trade-offs between verification accuracy and computational efficiency, demonstrating that lightweight verifiers can outperform larger LLM-based scorers while requiring orders-of-magnitude fewer FLOPs. Query x\in\mathcal{X} Reference R(x) Response Generator G Output y Conformal Factuality Framework Calibration Data \widetilde{\mathcal{X}} Filtered Output y^{\prime} Input to LLM Conformal Filtering Figure 1: Overview of our framework. Given a query x and retrieved references R(x) , the Response Generator G produces an output y . The conformal factuality framework utilizes a separate calibration data to determine a threshold used to filter out information from the output y and yield y^{\prime} . (See Figure 2 for the details of different stages involved in conformal filtering.) 2 Problem Setting, Datasets, Models, and Metrics In this section, we describe the conformal filtering framework for RAG-based LLMs (Figure 1 ), introduce the scoring functions, datasets and evaluation metrics used in this work. Let x\in\mathcal{X} denote an input query. Let R(x) denote the reference material sufficient to answer x . We assume the existence of an oracle retriever that can retrieve R(x) such that the true answer y^{\star} is either in or can be deduced from R(x) . This enables us to focus on evaluating the effectiveness of conformal filtering of answers generated by RAG-based LLMs by decoupling the effects of specific methods used for retrieval. A response generator G , instantiated as a large language model (LLM), is prompted with both x and R(x) to produce an output y=G(x,R(x)) . The goal of the conformal filtering is to create a final

Read the complete text