Improving Neural Topic Modeling with Semantically-Grounded Soft Label Distributions

Raymond Li, Amirhossein Abaskohi, Chuyuan Li, Gabriel Murray, Giuseppe Carenini · 2026

Read Improving Neural Topic Modeling with Semantically-Grounded Soft Label Distributions by Raymond Li, Amirhossein Abaskohi, Chuyuan Li, Gabriel Murray, Giuseppe Carenini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-17907.txt.

Description

Raymond Li and team improve topic modeling by distilling soft labels from language models. Traditional neural topic models optimize by copying bag-of-words codes, overlooking context information and struggling with data thinness. The new training frame, Distilling Soft Labels, constructs context-enriched copying signals. It maps next token odds, given a special prompt, onto a set word list. It trains topic models to copy soft labels using language model hidden states.

That approach yields higher-quality topics more closely aligned with the underlying theme structure of the corpus. Tests show large gains in topic clarity and sorting scores over existing baselines. A search-based metric introduced by the authors shows the approach clearly beating existing methods at identifying similar texts in meaning, highlighting power for search-focused uses. Readers gain a topic modeling method that uses language model knowledge without requiring full language model fine-tuning.

Catalog record

CREATORRaymond Li, Amirhossein Abaskohi, Chuyuan Li, Gabriel Murray, Giuseppe Carenini
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-17907

Browse the full library · About the Tower of Babel