Understanding New-Knowledge-Induced Factual Hallucinations in LLMs: Analysis, Solution and Interpretation
Renfei Dang, Peng Hu, Changjiang Gao, Shujian Huang · 2025
Read Understanding New-Knowledge-Induced Factual Hallucinations in LLMs: Analysis, Solution and Interpretation by Renfei Dang, Peng Hu, Changjiang Gao, Shujian Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-02626.txt.
Introduction
Renfei Dang, Peng Hu, Zhejian Lai, Changjiang Gao, Min Zhang, and Shujian Huang investigate a side effect of teaching language models new facts: they start hallucinating about old ones. Fine-tuning on fresh knowledge provokes confident errors on previously mastered material, but how the damage spreads was unclear. The team built a controlled dataset mixing biography material with reasoning tasks, and analysed failures by knowledge type and task type.
Hallucinations hit not only tasks involving the new knowledge but propagated to unrelated evaluations. The strongest driver was not how much new material was added overall, but whether one whole knowledge type was unfamiliar: that pattern reliably raised hallucination rates. Interpretability analysis found the mechanism: learning new knowledge weakens attention to key entities in a question, leaving the model leaning on surrounding context. Mixing a little known knowledge back into late training restored the attention pattern and cut the hallucinations substantially.
Catalog record
| CREATOR | Renfei Dang, Peng Hu, Changjiang Gao, Shujian Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-02626
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Understanding New-Knowledge-Induced Factual Hallucinations in LLMs: Analysis, Solution and Interpretation Renfei Dang, Peng Hu, Changjiang Gao, Shujian Huang Abstract Previous studies show that introducing new knowledge during large language models (LLMs) fine-tuning can lead to the generation of erroneous output when tested on known information, thereby triggering factual hallucinations. However, existing studies have not deeply investigated the specific manifestations and underlying mechanisms of these hallucinations. Our work addresses this gap by designing a controlled dataset Biography-Reasoning , and conducting a fine-grained analysis across multiple knowledge types and two task types, including knowledge question answering (QA) and knowledge reasoning tasks. We find that when fine-tuned on a dataset in which a specific knowledge type consists entirely of new knowledge, LLMs exhibit significantly increased hallucination tendencies. This suggests that the high unfamiliarity of a particular knowledge type, rather than the overall proportion of new knowledge, is a stronger driver of hallucinations, and these tendencies can even affect other knowledge types in QA tasks. To mitigate such factual hallucinations, we propose KnownPatch, which patches a small number of known knowledge samples in the later stages of training, effectively alleviating new-knowledge-induced hallucinations. Through attention analysis, we find that learning new knowledge reduces the model’s attention to key entities in the question, thus causing excessive focus on the surrounding context, which may increase the risk of hallucination. Moreover, the attention pattern can propagate to similar contexts, facilitating the spread of hallucinations to textually similar questions. Our method effectively mitigates the disruption of new knowledge learning to the model’s attention on key entities, accompanied by improved performance. Code — https://github.com/NJUNLP/New-Knowledge-Induced-Factual-Hallucinations 1 Introduction LLMs embed rich factual knowledge in their parameters during pre-training on massive text corpora ( Petroni et al. 2019 ; Cohen et al. 2023 ) . Subsequently, post-training enables them to learn to follow human instructions and exhibit superior performance across various downstream tasks ( Ouyang et al. 2022 ; Wei et al. 2022 ) . Figure 1: The impact of learning new knowledge on attention patterns and hallucination behavior. When an model is trained on unknown facts, it may be more prone to produce factual hallucinations, the severity of which correlates with the degree of attention paid to key entities and is modulated by contextual similarity. By injecting a small amount of known knowledge at the end of training via KnowPatch, this issue can be effectively mitigated. However, during the Supervised Fine-Tuning (SFT) phase, models may encounter new knowledge not covered in pre-training. Prior researches ( Ghosal, Hashimoto, and Raghunathan 2024 ; Lin et al. 2023 ; Ovadia et al. 2023 ; Gekhman et al. 2024 ; Sun et al. 2025 ) suggest that introducing new knowledge in the post-training phase increases the risk of factual hallucinations, where models generate fabricated yet plausible statements. This occurs because, when models learn new facts absent from pre-training, they may erroneously generate related information in irrelevant contexts ( Gekhman et al. 2024 ; Sun et al. 2025 ) . These studies primarily focus on the effects within knowledge-intensive QA tasks during SFT, and we advance this line of research by conducting a systematic analysis of fine-grained manifestations and underlying causes of hallucinations. To enable a comprehensive investigation, we construct a controlled experimental dataset, Biography-Reasoning . The dataset is composed of biographical entities and their four attributes (birth, death, major, and university), which serve as four knowledge types. We further design twelve reasoning tasks using these knowledge. By controlling the proportion of known and unknown knowledge within different types and tasks in the training data, we systematically investigate the impact of learning new knowledge on hallucination risks. Our experiments reveal that, for knowledge QA tasks, training on unknown knowledge significantly elevates hallucination risks in the same type test set, with some cross-type influence on other QA test sets; in tasks involving knowledge reasoning, training on reasoning tasks containing unknown knowledge primarily affects the same reasoning task, with some impact on knowledge QA and limited influence on other reasoning tasks. Importantly, we find that if a knowledge type consists entirely of new knowledge, even a small amount of such data can markedly increase hallucination tendencies. This poses challenges for non-perfect filtering-based hallucination mitigation methods: inadvertently retained small amounts of unknown data can trigger severe hallucinations. To address this challenge, we propose KnownPatch , a simple yet effective strategy that places a small number of known knowledge in later stages of training. Instead of relying on exhaustive filtering of unknown content, our method stabilizes learning by reinducing known knowledge late in training. Experiments show that even a small injection of known data significantly reduces hallucination tendencies across diverse settings. Finally, we perform an interpretability analysis of the model’s attention distribution. The results reveal that learning new knowledge significantly weakens the model’s attention to key entities in the question, instead may erroneously bind new knowledge to other contexts, thereby triggering factual hallucinations. Therefore, tasks that share similar contexts are more likely to be affected and exhibit stronger hallucination tendencies. KnownPatch effectively restores and enhances the model’s attention to key entities, thereby mitigating hallucination risks. These findings are visually presented in Figure 1 . In summary, the main contributions of this paper are: • Fine-Grained Analysis : A detailed analysis across knowledge types and task types reveals the manifestations of new-knowledge-induced hallucinations, showing that when all knowledge within a specific type is entirely unknown, it is more likely to trigger severe hallucinations, even on unrelated QA test sets. • Mitigation Strategy : The introduction of KnownPatch, a training method that places a small amount of known knowledge in the later stages, significantly reducing hallucination risks without data modifications. • Mechanism Interpretability : An analysis of attention mechanisms shows that learning new knowledge reduces attention to key question entities and erroneously binds new knowledge to contexts, causing hallucinations. Moreover, similar contexts facilitate the spread of these altered attention patterns, enabling cross-task hallucination effects. 2 Related Work New Knowledge and Hallucinations Existing studies have indicated that introducing new knowledge into LLMs may trigger hallucinations ( Ghosal, Hashimoto, and Raghunathan 2024 ; Lin et al. 2023 ; Ovadia et al. 2023 ) . Subsequent works have provided deeper analyses of this phenomenon ( Gekhman et al. 2024 ; Kang et al. 2024 ; Sun et al. 2025 ) . Gekhman et al. 2024 found that as the proportion of new knowledge in fine-tuning data increases, the model’s hallucination tendency intensifies. Kang et al. 2024 analyzed that when fine-tuned LLMs encounter unknown queries during testing, their responses imitate those associated with unknown examples in the fine-tuning data. Sun et al. 2025 , from the perspective of token probabilities, examined how, after learning new knowledge, the generation probability of relevant answer entity tokens in irrelevant contexts increases significantly, suggesting that the model may erroneously generalize knowledge, leading to hallucinations. However, previous studies focus mainly on closed-book QA settings with mixed knowledge types during training, while our controlled setup disentangles them to provide a more detailed analysis of new knowledge-induced hallucinations across types and tasks. Furthermore, we also investigate the underlying mechanisms of these phenomenon through an analysis of attention weights. Reducing Hallucinations Numerous studies are currently exploring ways to mitigate model hallucinations. A common approach involves