The Curious Case of Factuality Finetuning: Models’ Internal Beliefs Can Improve Factuality

Benjamin Newman, Abhilasha Ravichander Jaehun Jung, Rui Xin, Hamish Ivison, Yegor Kuznetsov, Pang Wei Koh, Yejin Choi · 2025

Read The Curious Case of Factuality Finetuning: Models’ Internal Beliefs Can Improve Factuality by Benjamin Newman, Abhilasha Ravichander Jaehun Jung, Rui Xin, Hamish Ivison, Yegor Kuznetsov, Pang Wei Koh, Yejin Choi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08371.txt.

Introduction

Benjamin Newman and co-authors study a surprising route to more factual language models: train them on data they themselves believe. Their 2025 paper tackles hallucination, where models state false things as fact. Training on gold factual data can help, but such data are costly, and correct yet unfamiliar data may even raise later hallucination.

Across tests, training on model generated data that the model judged factual beat training on gold data alone. Filtering either kind of data by the model's own judgments helped most when paired with model generated text. Gains held across three domains. The authors conclude that a model's internal beliefs give a strong signal for factuality work.

Catalog record

CREATORBenjamin Newman, Abhilasha Ravichander Jaehun Jung, Rui Xin, Hamish Ivison, Yegor Kuznetsov, Pang Wei Koh, Yejin Choi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE77 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08371

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Curious Case of Factuality Finetuning: Models’ Internal Beliefs Can Improve Factuality Benjamin Newman, Abhilasha Ravichander Jaehun Jung, Rui Xin, Hamish Ivison, Yegor Kuznetsov, Pang Wei Koh, Yejin Choi Abstract Language models are prone to hallucination —generating text that is factually incorrect. Finetuning models on high-quality factual information can potentially reduce hallucination, but concerns remain; obtaining factual gold data can be expensive and training on correct but unfamiliar data may potentially lead to even more downstream hallucination. What data should practitioners finetune on to mitigate hallucinations in language models? In this work, we study the relationship between the factuality of finetuning data and the prevalence of hallucinations in long-form generation tasks. Counterintuitively, we find that finetuning on factual gold data is not as helpful as finetuning on model-generated data that models believe to be factual. Next, we evaluate filtering strategies applied on both factual gold data and model-generated data, and find that finetuning on model-generated data that is filtered by models’ own internal judgments often leads to better overall factuality compared to other configurations: training on gold data filtered by models’ judgments, training on gold data alone, or training on model-generated data that is supported by gold data. These factuality improvements transfer across three domains we study, suggesting that a models’ own beliefs can provide a powerful signal for factuality. 1 1 1 github.com/bnewm0609/epistemic-training 1 Introduction A significant obstacle limiting widespread use of language models (LMs) is model hallucination—models tend to generate text that is factually incorrect without any indication to readers ( Wang et al. 2024 ) . This problem is particularly acute in long-form generation tasks, where generations can be partially correct ( Huang et al. 2024b ) , and factual errors might accumulate during generation ( Zhang et al. 2024b ) . As a result, finetuning models to improve factuality has become a standard component of post-training recipes ( Dubey et al. 2024 ; Yang et al. 2023 ; Zhang et al. 2024a ) . However, this can be challenging—recent work in short-form question answering has noted that finetuning on factually correct but unfamiliar knowledge can actually hurt downstream factuality; accordingly, these works recommend finetuning on knowledge that is both factually correct and familiar ( Gekhman et al. 2024 ; Ghosal et al. 2024 ) . Figure 1: We manipulate two aspects of finetuning data: the knowledge source (using the gold document directly or employing an LM to generate a document based on the entity discussed in the gold document) and filtering method (entailment from external documents or probes on model hidden states) used for curating training data to improve finetuned model factuality. Epistemic Training (bolded)—training on generated data, filtered with internal knowledge probes (internal) outperforms both training on gold data as well as generated data filtered with entailment to gold documents (external). In this work, we find that model factuality improves even when finetuning on data that is less factual —in the settings we study, factuality is often improved most when the finetuning data aligns with models’ internal knowledge, even if the knowledge is unsupported by gold documents. We come to this conclusion by manipulating two attributes of finetuning data in controlled long-form generation experiments (outlined in Figure 1 ). First we consider the knowledge source used to create the finetuning data, and we compare finetuning on gold data to training on the models’ own generations. While the first is more factual, we find that finetuning on the latter leads to greater improvements in factuality when describing a held out set of entities. Second, we consider how to filter the training data. Long form generations may contain incorrect claims, so we filter claims from the generations before training on them. We compare filtering using external knowledge (only keeping information entailed by gold documents) to filtering based on internal knowledge (only keeping facts that a probe predicts are true from models’ hidden states). We use this latter technique to represent what models believe to be true, and it is motivated by prior work indicating that probes trained on model representations can predict whether models’ own generations are factual ( Azaria & Mitchell 2023 ; Liu et al. 2023 ; Joshi et al. 2024 ) . Compared to training on data filtered with external knowledge, using internal knowledge filters often leads to a greater increase in factuality on descriptions of held-out entities, despite the training data being less factual. Based on these observations, we introduce Epistemic Training , a procedure to post-train models using data generated by, and filtered using, the model’s own internal knowledge—the best performing combination of knowledge source and filter. We use Epistemic Training to analyze the dependence of models on the factuality of their training data and find that it increases factuality compared to training on gold data alone by up between 12–30 percentage points (Figure 1 top-most versus bottom-most columns). Finally, while Epistemic Training requires less gold data compared to other methods, it still requires a small amount for training the internal knowledge filter. As such, we also investigate whether even this gold data is necessary by studying cross domain transfer—whether models finetuned with Epistemic Training on one domain can perform well on others. In general, we find evidence that factuality does generalize across domains although using in-domain data when possible is better. We summarize our contributions as follows: 1. We extend the findings that models trained on gold data are less factual than models trained on generated data ( Gekhman et al. 2024 ; Ghosal et al. 2024 ) to long-form generation settings. 2. We find that filtering claims using probes on models’ hidden states improves factuality over filtering using using entailment to external documents. 3. We find that the factuality gains learned on one domain can transfer to improvements on others. Our analysis sheds light on how model practitioners should filter data for post-training, as well as provides an understanding of specific signals that models can leverage to improve their factual accuracy. 2 Approach In order to investigate how the factuality of training data affects models’ hallucination rates, we compare different methods for creating finetuning data using different knowledge sources and factuality filters. Our data creation pipeline has three stages. First, we choose a knowledge source : selecting gold training documents or generating synthetic documents to create an initial training set. Second we choose a filtering strategy to remove incorrect claims from those documents. Finally we merge the remaining claims together to construct a final training set. We finetune models on this data and measure their hallucination rates. See Figure 1 for an overview. 2.1 Knowledge Source We compare training on two knowledge sources: gold documents, D_{\text{gold}} , and model-generated documents, D_{\text{gen}} . Formally, we consider a set of entities E , and documents D_{\text{gold}} where each entity e_{k} has an associated human-authored gold document, d_{k}\in D_{\text{gold}} , that contains a large amount of factual information about e_{k} . (Note that in our setting we have a separate E and D_{\text{gold}} for training and evaluation.) Generated Documents ( D_{\text{gen}} ). Our data for D_{\text{gen}} is generated as follows: We place each e_{k} in a prompt template and sample 10 generations per prompt, y^{i}_{k} , where i is the generation index. 2

Read the complete text