Exploring the Knowledge Mismatch Hypothesis: Hallucination Propensity in Small Models Fine-tuned on Data from Larger Models
Phil Wee, Riyadh Baghdadi · 2024
Read Exploring the Knowledge Mismatch Hypothesis: Hallucination Propensity in Small Models Fine-tuned on Data from Larger Models by Phil Wee, Riyadh Baghdadi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-00878.txt.
Description
Abstract Recently, there has been an explosion of large language models created through fine-tuning with data from larger models. These small models able to produce outputs that appear qualitatively similar to significantly larger models. However, one of the key limitations that have been observed with these models is their propensity to hallucinate significantly more often than larger models.
Catalog record
| CREATOR | Phil Wee, Riyadh Baghdadi |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-00878
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exploring the Knowledge Mismatch Hypothesis: Hallucination Propensity in Small Models Fine-tuned on Data from Larger Models Phil Wee, Riyadh Baghdadi Abstract Recently, there has been an explosion of large language models created through fine-tuning with data from larger models. These small models able to produce outputs that appear qualitatively similar to significantly larger models. However, one of the key limitations that have been observed with these models is their propensity to hallucinate significantly more often than larger models. In particular, they have been observed to generate coherent outputs that involve factually incorrect information and spread misinformation, toxicity, and stereotypes. There are many potential causes of hallucination, of which, one hypothesis is that fine-tuning a model on data produced by a larger model leads to a knowledge mismatch which contributes to hallucination. In particular, it is hypothesized that there is a mismatch between the knowledge that is fed to the model to fine-tune it and the knowledge that is already present in the graph. Fine-tuning the model on data that has such mismatch could contribute to an increased propensity to hallucinate. We show that on an unseen test set, a smaller model fine-tuned on data generated from a larger model produced more wrong answers when compared to models fine-tuned on data created by the small model, which confirms the hypothesis. I Introduction A well-known limitation of large language models is their ability to hallucinate or generate factually incorrect statements [ 16 ] . While large language models are often able to appear fluent, the responses generated by these systems have been observed to often produce statements that are misleading, factually incorrect and harmful. In the context of dialogue-based systems, it has been observed that models often produce responses that are not supported by the evidence available to the system [ 19 ] . In the context of generative question and answering systems, models have been observed to provide factually incorrect responses [ 18 ] . This is a key issue in the field of Large Language Models because of the potential harm that hallucination can pose to users. For instance, in medical applications, a medical summary generated from patient information could pose a risk to the patient if the generated outputs involve hallucination. In this case, a factually false recommendation generated by the model can lead to a life-threatening incident for the patient. In recent times, this is even more concerning because of the rise of strong large language models that are often able to produce coherent and semantically convincing responses while at the same time, involving hallucination in their outputs. It has become increasingly difficult to tell the difference between the outputs of a model and text written by humans, especially in more general and generic topics and in prompts with short responses. This furthers the risk of hallucination in the outputs of models - the high level of coherence in the outputs of many recent models can easily fool many humans and even many experts. A key development in the field of open source models is the explosion of large language models created through fine-tuning with data from larger models. [ 10 ] [ 3 ] [ 9 ] [ 12 ] [ 11 ] Most of these fine-tuned models are not only relatively small and easy to reproduce but are also relatively performant and are able to produce outputs that appear qualitatively similar to significantly larger models. However, one key limitation that many of these models share is their propensity to hallucinate significantly more often than larger models. While there has been a lot of previous work in addressing hallucination through further tuning via techniques such as Reinforcement Learning through Human Feedback (RLHF) [ 14 ] [ 15 ] and Reinforcement Learning through AI Feedback (RLAI) [ 13 ] , there has been relatively little work in analyzing the role of underlying training techniques such as fine-tuning. Hallucination has a lot of possible reasons, both with respect to data and the model. Our goal in this paper is to study one of the possible reasons: fine-tuning data. In particular, we want to verify the hypothesis that fine-tuning a model on data produced by a larger model leads to a knowledge mismatch, which may contribute to hallucination. Our analysis finds that on an unseen test set, a smaller model fine-tuned on data generated from a larger model produced more wrong answers when compared to models fine-tuned on data created by the small model, which confirms the hypothesis. Our findings question the overall effectiveness of current fine-tuning practices, given the potential for a knowledge mismatch that such fine-tuning may cause and its implications on model hallucination. II Origins of Hallucination Hallucination is observed to have many potential origins and is observed to be rooted in both data and models [ 16 ] [ 18 ] . With respect to data, factually inaccurate data is known to be a key driver of hallucination - if the answer that the model was trained on was wrong, it is likely going to produce a wrong answer. However, even if the training data is factually accurate, hallucination can also still occur as there is simply no way to fully cover every single possible question in the training data. As a result, for questions that it has not seen, models are forced to generalize responses without being able to verify where it is accurate, relevant, or appropriate, which also contributes to hallucination. This issue also applies to other forms of problematic data, such as stereotypical data and biased data - due to the scale of the data used in the training of foundational models, it is impossible to fully or even significantly filter through and clean the training data. As a result, these issues rooted in problematic training data is likely to be present in many foundational models, which most fine-tuned model are based on and inherit from. Another key reason source of hallucination is the prescence of duplicates in the training data [ 22 ] , for which it is common for many training corpora to contain near-duplicate examples and long repetitive substrings. Not only does this lead to wasted compute resources in training, but it is observed to the model memorizing the common snippets. This can contribute to hallucination as the increased propensity to output memorized snippets may lead to the model generating them even in situations where it is not appropriate or factual to do so. Furthermore, Ji et al. [ 18 ] note that large language models exhibit Innate Divergence. This is because by nature, some natural language generation (NLG) tasks do not always have factual knowledge alignment between the source input text and the target reference, especially in situations that value diversity in the generated output. For instance, it may be acceptable for open-domain dialogue systems to respond in a subjective style, as this can make generated dialogues more engaging and diverse. However, it has been observed that such dataset characteristics can lead to hallucinations. Such divergence can also lead to issues in the other direction, that is, the training data may contain pieces of information that are relatively informal and subjective, which a model may use in data generation