Semantic Self-Distillation for Language Model Uncertainty

Edward Phillips, Sean Wu, Boyan Gao, David A. Clifton · 2026

Read Semantic Self-Distillation for Language Model Uncertainty by Edward Phillips, Sean Wu, Boyan Gao, David A. Clifton (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04577.txt.

Description

Edward Phillips and 4 co-authors tackle a practical problem in a 2026 paper: knowing when a language model is unsure. Sampling many answers and measuring how their meanings spread works, but it is too slow for time-sensitive uses.

Their technique, Semantic Self-Distillation, trains a small student model to predict that spread before the main model answers. The student's predicted distribution yields an uncertainty signal from its entropy, a measure of spread. It also allows judging single answers by their probability. On trivia and general knowledge tests, the student rivaled the costly sampling method at predicting hallucinations. It also helped detect out-of-domain prompts and choose among multiple-choice answers.

Catalog record

CREATOREdward Phillips, Sean Wu, Boyan Gao, David A. Clifton
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-04577

Browse the full library · About the Tower of Babel