Improving Neural Topic Modeling with Semantically-Grounded Soft Label Distributions
Raymond Li, Amirhossein Abaskohi, Chuyuan Li, Gabriel Murray, Giuseppe Carenini · 2026
Read Improving Neural Topic Modeling with Semantically-Grounded Soft Label Distributions by Raymond Li, Amirhossein Abaskohi, Chuyuan Li, Gabriel Murray, Giuseppe Carenini (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-17907.txt.
Description
Raymond Li and team improve topic modeling by distilling soft labels from language models. Traditional neural topic models optimize by copying bag-of-words codes, overlooking context information and struggling with data thinness. The new training frame, Distilling Soft Labels, constructs context-enriched copying signals. It maps next token odds, given a special prompt, onto a set word list. It trains topic models to copy soft labels using language model hidden states.
That approach yields higher-quality topics more closely aligned with the underlying theme structure of the corpus. Tests show large gains in topic clarity and sorting scores over existing baselines. A search-based metric introduced by the authors shows the approach clearly beating existing methods at identifying similar texts in meaning, highlighting power for search-focused uses. Readers gain a topic modeling method that uses language model knowledge without requiring full language model fine-tuning.
Catalog record
| CREATOR | Raymond Li, Amirhossein Abaskohi, Chuyuan Li, Gabriel Murray, Giuseppe Carenini |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 56 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-17907