Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillation

Hieu Nguyen, Zihao He, Shoumik Atul Gandre, Ujjwal Pasupulety, Sharanya Kumari Shivakumar, Kristina Lerman · 2025

Read Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillation by Hieu Nguyen, Zihao He, Shoumik Atul Gandre, Ujjwal Pasupulety, Sharanya Kumari Shivakumar, Kristina Lerman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11306.txt.

Description

Hieu Nguyen and five co-authors curb hallucination in large language models with smoothed knowledge distillation. Models often state false or ungrounded claims, which bars them from high stakes use. Their work came out in twenty twenty-five.

Hard labels in training push fixed answers, breed overconfidence, and ignore the doubt built into language. In their fix, a teacher model gives soft labels to a student during fine tuning on teaching data. That softening trims overconfidence and firms factual grounding. Tests on summary tasks across model families cut hallucination versus standard fine tuning, while general task scores held, the authors report.

Catalog record

CREATORHieu Nguyen, Zihao He, Shoumik Atul Gandre, Ujjwal Pasupulety, Sharanya Kumari Shivakumar, Kristina Lerman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11306

Browse the full library · About the Tower of Babel