Evaluating Negative Sampling Approaches for Neural Topic Models
Suman Adhya, Avishek Lahiri, Debarshi Kumar Sanyal, Partha Pratim Das · 2025
Read Evaluating Negative Sampling Approaches for Neural Topic Models by Suman Adhya, Avishek Lahiri, Debarshi Kumar Sanyal, Partha Pratim Das (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18167.txt.
Introduction
Suman Adhya, Avishek Lahiri, Debarshi Kumar Sanyal, and Partha Pratim Das test whether negative sampling helps neural topic models learn better topics, in a paper from 2025. Topic models sort large text sets into themes with no labels given. Negative sampling, which trains a model by contrast with wrong examples, had not been well tested in this label-free field.
The team added negative sampling to the decoder of several autoencoder-based topic models. Tests ran on four public data sets. Topics grew more coherent and more diverse, and document sorting improved as well. Human raters also judged the topics to be of higher quality. The authors present the trick as a cheap way to lift topic model performance.
Catalog record
| CREATOR | Suman Adhya, Avishek Lahiri, Debarshi Kumar Sanyal, Partha Pratim Das |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-18167
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Evaluating Negative Sampling Approaches for Neural Topic Models Suman Adhya, Avishek Lahiri, Debarshi Kumar Sanyal, Partha Pratim Das Abstract Negative sampling has emerged as an effective technique that enables deep learning models to learn better representations by introducing the paradigm of “learn-to-compare”. The goal of this approach is to add robustness to deep learning models to learn better representation by comparing the positive samples against the negative ones. Despite its numerous demonstrations in various areas of computer vision and natural language processing, a comprehensive study of the effect of negative sampling in an unsupervised domain like topic modeling has not been well explored. In this paper, we present a comprehensive analysis of the impact of different negative sampling strategies on neural topic models. We compare the performance of several popular neural topic models by incorporating a negative sampling technique in the decoder of variational autoencoder based neural topic models. Experiments on four publicly available datasets demonstrate that integrating negative sampling into topic models results in significant enhancements across multiple aspects, including improved topic coherence, richer topic diversity, and more accurate document classification. Manual evaluations also indicate that the inclusion of negative sampling into neural topic models enhances the quality of the generated topics. These findings highlight the potential of negative sampling as a valuable tool for advancing the effectiveness of neural topic models. † † impactstatement: With the rapid advancement of technology, there has been a significant increase in the availability of text documents in digital format. Categorizing these documents based on their underlying content is crucial to facilitate easy access for users. However, manual labeling of these documents with their corresponding domain tags can be laborious and time-consuming due to the large volume of the corpus. Topic modeling techniques have emerged as a valuable tool in this context, as they can extract latent topics from a large corpus and label the documents with their dominant topics in an unsupervised manner. While traditional models pose computational challenges, neural topic models offer enhanced flexibility and scalability. Negative sampling-based models emphasize learning the document similarities and distinctions, thereby improving the quality of the learned topics. This paper conducts an empirical evaluation of several neural topic models based on negative sampling, implemented within a standardized framework to facilitate reproducibility. Thus, it will assist researchers and practitioners in identifying the most effective negative sampling strategies that may be employed for improving topic models. I Introduction In topic modeling, latent variables referred to as topics are described as distributions over words. Given a corpus, a document is modeled as a random mixture over topics. The objective of topic modeling is to compute the posterior distribution of topics with the aim of reconstructing the given input document. However, determining the exact posterior is often impractical [ 1 ] . Consequently, various inferencing algorithms are employed to approximate the true posterior distribution. Nonetheless, these methods encounter a notable drawback due to their high computational costs and the necessity for re-derivation of the inference method when minor alterations occur in the modeling assumptions. To tackle these challenges, Miao et al. [ 2 ] introduced a neural topic model grounded in the Variational Autoencoder (VAE) framework. Recent advancements in this field have focused on incorporating diverse distributions on the document-topic distributions, integrating word embeddings, contextual information from documents, and utilizing available metadata from texts [ 3 , 4 , 5 , 6 , 7 , 8 , 9 ] . Another approach to improve the performance of topic models is negative sampling: it introduces noise during the training process to enhance the model’s robustness to out-of-distribution samples and mitigate the representation collapse problem [ 10 , 11 ] . Recently, we proposed a negative sampling-based method integrated with a contextualized neural topic model [ 12 ] , that generates document-topic vectors as well as their perturbed forms, given a corpus, and uses a triplet loss so that the document reconstructed from the correct document-topic vector is similar to the input document but dissimilar to the one reconstructed from the perturbed vector. However, this negative sampling methodology is quite generic and can be integrated with other neural topic models. In this paper, we perform this integration and analyze the performance of seven high-performance neural topic models enhanced with the above negative sampling strategy 1 1 1 Code is available at https://github.com/AdhyaSuman/Eval_NegTM . To make a comprehensive analysis, we compare the augmented models not only with their original versions but also with existing topic models that use contrastive learning. We observe that on most of the datasets that we experimented with, the performance, in terms of topic coherence and diversity, improves when a topic model is enhanced with our negative sampling strategy. Our contributions can be summarized as follows: • We incorporate a negative sampling strategy into seven neural topic models, and assess its effectiveness and robustness. • We bring together a diverse set of neural topic models into a unified framework that categorizes the models based on the negative sampling strategy employed in them. • We present a comprehensive analysis of the above topic models assessing the influence of negative sampling through quantitative, qualitative, and task-specific evaluations. Additionally, we visualize the latent spaces generated by these models to assess their ability to disentangle the latent representations of documents. • Our assessments demonstrate that topic models incorporating negative sampling surpass those without this strategy for almost all datasets and experimental settings considered. The visualizations also highlight a discernible enhancement in disentanglement ability when negative sampling is incorporated. The remaining sections of the paper are organized as follows. In Section II , we provide an overview of the evolution of neural topic modeling, highlighting works related to the scope of this paper. Section III presents a comprehensive description of existing negative sampling approaches, conducting a detailed analysis of their methodological components and associated objective functions. Section IV contains the details on the datasets and model configurations used in evaluations. The empirical investigations are detailed in Section V , where we describe into the specifics of experimental configurations, evaluation methodologies, and the conclusions drawn from these empirical inquiries. Moving forward, Section VI presents an evaluation of the impact of different vocabulary sizes on our proposed negative sampling approach, and how the modifications due to the incorporation of negative sampling strategy to the models affect the training time of the models. Additionally, in Section VII , we provide a comparative analysis of our method with a recently proposed LLM-based topic model. The paper concludes in Section VIII . II Related Work Topic modeling assumes that a document can be represented as a distribution over topics, where each topic is a distribution over the entire vocabulary. A popular approach for topic modeling is Latent Dirichlet Allocation (LDA) [ 13 ] in which the document-topic distribution is modeled as a Dirichlet distribution. However, determining the exact posterior distribution of the hidden variables in LDA is impractical [ 1 ] . Various approximate inference methods have been proposed to approximately determine these distribution parameters, such as Gibbs sampling [ 14 ] , Expectation-Maximization (EM) [ 15 ] , and Variational Bayes inference [ 13 ] . Gibbs sampling, a Markov chain Monte Carlo algorithm, aids in statistical