Evaluating Negative Sampling Approaches for Neural Topic Models Suman Adhya, Avishek Lahiri, Debarshi Kumar Sanyal, Partha Pratim Das Abstract Negative sampling has emerged as an effective technique that enables deep learning models to learn better representations by introducing the paradigm of “learn-to-compare”. The goal of this approach is to add robustness to deep learning models to learn better representation by comparing the positive samples against the negative ones. Despite its numerous demonstrations in various areas of computer vision and natural language processing, a comprehensive study of the effect of negative sampling in an unsupervised domain like topic modeling has not been well explored. In this paper, we present a comprehensive analysis of the impact of different negative sampling strategies on neural topic models. We compare the performance of several popular neural topic models by incorporating a negative sampling technique in the decoder of variational autoencoder based neural topic models. Experiments on four publicly available datasets demonstrate that integrating negative sampling into topic models results in significant enhancements across multiple aspects, including improved topic coherence, richer topic diversity, and more accurate document classification. Manual evaluations also indicate that the inclusion of negative sampling into neural topic models enhances the quality of the generated topics. These findings highlight the potential of negative sampling as a valuable tool for advancing the effectiveness of neural topic models. † † impactstatement: With the rapid advancement of technology, there has been a significant increase in the availability of text documents in digital format. Categorizing these documents based on their underlying content is crucial to facilitate easy access for users. However, manual labeling of these documents with their corresponding domain tags can be laborious and time-consuming due to the large volume of the corpus. Topic modeling techniques have emerged as a valuable tool in this context, as they can extract latent topics from a large corpus and label the documents with their dominant topics in an unsupervised manner. While traditional models pose computational challenges, neural topic models offer enhanced flexibility and scalability. Negative sampling-based models emphasize learning the document similarities and distinctions, thereby improving the quality of the learned topics. This paper conducts an empirical evaluation of several neural topic models based on negative sampling, implemented within a standardized framework to facilitate reproducibility. Thus, it will assist researchers and practitioners in identifying the most effective negative sampling strategies that may be employed for improving topic models. I Introduction In topic modeling, latent variables referred to as topics are described as distributions over words. Given a corpus, a document is modeled as a random mixture over topics. The objective of topic modeling is to compute the posterior distribution of topics with the aim of reconstructing the given input document. However, determining the exact posterior is often impractical [ 1 ] . Consequently, various inferencing algorithms are employed to approximate the true posterior distribution. Nonetheless, these methods encounter a notable drawback due to their high computational costs and the necessity for re-derivation of the inference method when minor alterations occur in the modeling assumptions. To tackle these challenges, Miao et al. [ 2 ] introduced a neural topic model grounded in the Variational Autoencoder (VAE) framework. Recent advancements in this field have focused on incorporating diverse distributions on the document-topic distributions, integrating word embeddings, contextual information from documents, and utilizing available metadata from texts [ 3 , 4 , 5 , 6 , 7 , 8 , 9 ] . Another approach to improve the performance of topic models is negative sampling: it introduces noise during the training process to enhance the model’s robustness to out-of-distribution samples and mitigate the representation collapse problem [ 10 , 11 ] . Recently, we proposed a negative sampling-based method integrated with a contextualized neural topic model [ 12 ] , that generates document-topic vectors as well as their perturbed forms, given a corpus, and uses a triplet loss so that the document reconstructed from the correct document-topic vector is similar to the input document but dissimilar to the one reconstructed from the perturbed vector. However, this negative sampling methodology is quite generic and can be integrated with other neural topic models. In this paper, we perform this integration and analyze the performance of seven high-performance neural topic models enhanced with the above negative sampling strategy 1 1 1 Code is available at https://github.com/AdhyaSuman/Eval_NegTM . To make a comprehensive analysis, we compare the augmented models not only with their original versions but also with existing topic models that use contrastive learning. We observe that on most of the datasets that we experimented with, the performance, in terms of topic coherence and diversity, improves when a topic model is enhanced with our negative sampling strategy. Our contributions can be summarized as follows: • We incorporate a negative sampling strategy into seven neural topic models, and assess its effectiveness and robustness. • We bring together a diverse set of neural topic models into a unified framework that categorizes the models based on the negative sampling strategy employed in them. • We present a comprehensive analysis of the above topic models assessing the influence of negative sampling through quantitative, qualitative, and task-specific evaluations. Additionally, we visualize the latent spaces generated by these models to assess their ability to disentangle the latent representations of documents. • Our assessments demonstrate that topic models incorporating negative sampling surpass those without this strategy for almost all datasets and experimental settings considered. The visualizations also highlight a discernible enhancement in disentanglement ability when negative sampling is incorporated. The remaining sections of the paper are organized as follows. In Section II , we provide an overview of the evolution of neural topic modeling, highlighting works related to the scope of this paper. Section III presents a comprehensive description of existing negative sampling approaches, conducting a detailed analysis of their methodological components and associated objective functions. Section IV contains the details on the datasets and model configurations used in evaluations. The empirical investigations are detailed in Section V , where we describe into the specifics of experimental configurations, evaluation methodologies, and the conclusions drawn from these empirical inquiries. Moving forward, Section VI presents an evaluation of the impact of different vocabulary sizes on our proposed negative sampling approach, and how the modifications due to the incorporation of negative sampling strategy to the models affect the training time of the models. Additionally, in Section VII , we provide a comparative analysis of our method with a recently proposed LLM-based topic model. The paper concludes in Section VIII . II Related Work Topic modeling assumes that a document can be represented as a distribution over topics, where each topic is a distribution over the entire vocabulary. A popular approach for topic modeling is Latent Dirichlet Allocation (LDA) [ 13 ] in which the document-topic distribution is modeled as a Dirichlet distribution. However, determining the exact posterior distribution of the hidden variables in LDA is impractical [ 1 ] . Various approximate inference methods have been proposed to approximately determine these distribution parameters, such as Gibbs sampling [ 14 ] , Expectation-Maximization (EM) [ 15 ] , and Variational Bayes inference [ 13 ] . Gibbs sampling, a Markov chain Monte Carlo algorithm, aids in statistical inferencing. Though it is a simple and effective technique, it has a high computational cost. On the other hand, the EM algorithm iteratively calculates the expected log-likelihood and optimizes parameter values. Variational Bayes (VB) constructs a lower bound for data likelihood and selects parameters maximizing this bound. Monte Carlo techniques provide numerical approximations , whereas VB offers an exact analytical solution to the posterior approximation . Thus, computing the posterior is quite challenging. To handle some of these issues, the Neural Variational Document Model (NVDM) [ 2 ] introduces the AutoEncoding Variational Bayes (AEVB) [ 16 ] technique, commonly known as “black-box inference”, into topic modeling. It is built with a VAE, which comprises an encoder and a decoder. In a high-level description, the encoder takes an input vector and projects it into a lower-dimensional latent space, while the decoder reconstructs the input vector from this latent representation. During training, this model seeks to maximize the tractable evidence lower bound to maximize the likelihood of the data distribution. Successive works in this direction produced models such as the Gaussian Softmax (GSM) [ 3 ] , Autoencoding Variational Inference For Topic Models (AVITM) [ 4 ] , Sparse Contextual Hidden and Observed Language AutoencodeR (SCHOLAR) [ 5 ] , Embedded Topic Model (ETM) [ 6 ] , CombinedTM [ 7 ] and ZeroshotTM [ 8 ] . Negative sampling introduces a “learn-to-compare” paradigm [ 17 ] , employing an effective strategy for learning rich representations. This technique gained popularity in NLP tasks, notably in the word2vec algorithm [ 18 ] , to enhance training efficiency. In the domain of topic modeling, introducing negative samples in the VAE (encoder [ 19 ] and decoder [ 20 , 12 ] ) and employing adversarial learning [ 21 , 22 , 23 ] have been shown to increase the quality of the generated topics. The negative sampling-based technique [ 12 ] that we proposed only for the contextualized neural topic model [ 7 ] is a simple yet effective strategy, and we, therefore, integrate it with seven contemporary efficient neural topic models in this paper. We compare the enhanced topic models with their original versions (without negative sampling) and other related models proposed by researchers. Our previous work is thus significantly expanded in the current paper into a comprehensive evaluation of various negative sampling strategies in neural topic models. The OCTIS [ 24 ] and ToModAPI [ 25 ] frameworks, proposed recently, lack consideration for topic models with negative sampling. This paper utilizes OCTIS to introduce negative sampling into neural topic models, creating an open-source framework for fair model comparisons. This study addresses the existing gap in the literature by offering a detailed analysis of negative sampling in neural topic models. III Negative Sampling Strategies in Neural Topic Models Fig. 1: Topic model categories based on negative sampling techniques. Negative sampling techniques can assist neural topic models to learn better latent representations. We previously introduced a negative sampling strategy [ 12 ] , which involves perturbing the document-topic vector generated by the encoder and feeding it into the decoder. A triplet loss function ensures that the document reconstructed from the unperturbed topic vector is similar to the input document while being dissimilar to the document reconstructed from the perturbed vector. Thus, this technique effectively modifies the VAE decoder. However, the literature on topic modeling includes instances where alternative forms of contrastive learning have been applied. For instance, negative document samples might be directly input to the encoder, or an adversarial network architecture could replace the VAE model. To facilitate a comparative examination, we present a taxonomy that encompasses these diverse VAE-based neural topic models. In particular, we categorize them into four distinct groups as follows: (A) VAE-based neural topic models without negative sampling ( VAE-NTM ); (B) VAE-NTM with encoder negative sampling ( VAE-ENS ); (C) VAE-NTM with decoder negative sampling ( VAE-DNS ); (D) Adversarial network-based topic models ( Adv-Net ); TABLE I: Notations and terminologies. Symbol Description V Vocabulary size. T Number of topics. \boldsymbol{\mathbf{x}} Document representation vector of size V . (\boldsymbol{\mathbf{\mu}},\boldsymbol{\mathbf{\Sigma}}) Mean and covariance, respectively, in T -dimensional Gaussian. \boldsymbol{\mathbf{z}} Latent representation of size T . \sigma(\cdot) Softmax function. \boldsymbol{\mathbf{\theta}} Document-topic distribution vector of size T . \boldsymbol{\mathbf{\beta}} Topic-word matrix of dimension T\times V . \hat{\boldsymbol{\mathbf{x}}} Reconstructed document representation of size V . The topic models in each category are mentioned in Figure 1 . We will now describe each category in detail. Table I mentions the symbols that will be used in the description. III-A VAE-NTM This category includes VAE-based neural topic models without negative sampling. Within this category, a general technique is AVITM [ 4 ] , which incorporates a Laplace approximation to the Dirichlet prior [ 26 ] . It includes models like ProdLDA and NeuralLDA [ 4 ] . ProdLDA, by relaxing constraints on topic-word distributions, outperforms NeuralLDA. ETM [ 6 ] uses word2vec embeddings for robustness, while SCHOLAR [ 5 ] incorporates document metadata. The GSM model [ 3 ] employs a softmax function on the document-topic vector. In VAE-NTMs, the encoder takes a document representation vector ( \boldsymbol{\mathbf{x}} ) as input. This vector can be either a normalized bag-of-words vector ( \boldsymbol{\mathbf{x}}=\boldsymbol{\mathbf{x}}_{\BoW} ) or a term frequency-inverse document frequency (tf-idf) vector ( \boldsymbol{\mathbf{x}}=\boldsymbol{\mathbf{x}}_{\operatorname{TF-IDF}} ). For CombinedTM [ 7 ] , a contextualized representation of the document ( \boldsymbol{\mathbf{x}}_{c} ) is used alongside the bag-of-words representation ( \boldsymbol{\mathbf{x}}_{\BoW} ). On the other hand, ZeroShotTM [ 8 ] relies solely on the contextualized representation of the document as input, allowing for training in one language and testing in another. Given a Gaussian distribution as a prior and the input document representation vector \boldsymbol{\mathbf{x}} , the encoder returns the variational posterior q_{\boldsymbol{\mathbf{\phi}}}(\boldsymbol{\mathbf{z}}|\boldsymbol{\mathbf{x}}) , where \boldsymbol{\mathbf{\phi}} represents the encoder weights. We assume the variational posterior to be the Gaussian distribution \mathcal{N}\left(\boldsymbol{\mathbf{\mu}},\boldsymbol{\mathbf{\Sigma}}\right) , where \boldsymbol{\mathbf{\mu}} is the mean, and \boldsymbol{\mathbf{\Sigma}} represents the diagonal-covariance matrix. A latent representation \boldsymbol{\mathbf{z}} is stochastically sampled from the conditional distribution q_{\boldsymbol{\mathbf{\phi}}}(\boldsymbol{\mathbf{z}}|\boldsymbol{\mathbf{x}}) using the reparameterization trick [ 16 ] , given by \boldsymbol{\mathbf{z}}=\boldsymbol{\mathbf{\mu}}+\boldsymbol{\mathbf{\Sigma}}^{1/2}\odot\boldsymbol{\mathbf{\epsilon}} , where \odot denotes the Frobenius inner product and \boldsymbol{\mathbf{\epsilon}}\sim\mathcal{N}(\boldsymbol{\mathbf{0}},\boldsymbol{\mathbf{I}}) is random noise. In the decoder, this latent representation \boldsymbol{\mathbf{z}} serves as the logits of a softmax function to generate the document-topic distribution \boldsymbol{\mathbf{\theta}} for reconstructing the input document, \hat{\boldsymbol{\mathbf{x}}}=\Softmax(\boldsymbol{\mathbf{\beta}}^{\top}{\boldsymbol{\mathbf{\theta}}}) , where \boldsymbol{\mathbf{\beta}} denotes the decoder weights. The loss function captures both the reconstruction error ( \mathcal{L}_{\RL} ) and the KL-divergence ( \mathcal{L}_{\KL} ) between the prior and posterior of the latent distribution: \displaystyle\mathcal{L} \displaystyle=\mathcal{L}_{\RL}+\mathcal{L}_{\KL} (1) \displaystyle\equiv-\mathbb{E}_{\boldsymbol{\mathbf{z}}\sim q_{\boldsymbol{\mathbf{\phi}}}(\boldsymbol{\mathbf{z}}|\boldsymbol{\mathbf{x}})}[p_{\boldsymbol{\mathbf{\beta}}}(\boldsymbol{\mathbf{x}}|\boldsymbol{\mathbf{z}})]+\operatorname{D}_{\KL}\left(q_{\boldsymbol{\mathbf{\phi}}}(\boldsymbol{\mathbf{z}}|\boldsymbol{\mathbf{x}})\|p(\boldsymbol{\mathbf{z}})\right) A general framework of VAE-based neural topic models is illustrated in Figure 2 . Since the above VAE-based architecture uses a Gaussian prior (unlike LDA), AVITM [ 4 ] and models based on it have used a Laplace approximation to the Dirichlet. Fig. 2: A general framework for VAE-based topic models. III-B VAE-ENS CLNTM [ 19 ] is the only VAE-based topic model that integrates a negative sampling technique within its encoder. The framework is outlined in Figure 3 . In CLNTM, given an input document representation \boldsymbol{\mathbf{x}} , two additional samples are generated: a positive sample denoted by \boldsymbol{\mathbf{x}}^{+} and a negative sample denoted by \boldsymbol{\mathbf{x}}^{-} . These samples are converted to latent representations \boldsymbol{\mathbf{z}}^{+} and \boldsymbol{\mathbf{z}}^{-} , respectively, by the encoder. To optimize the encoder and promote better separation between these latent representations, CLNTM employs the InfoNCE loss [ 27 ] : \displaystyle\mathcal{L}_{\operatorname{InfoNCE}}=\mathbb{E}_{\boldsymbol{\mathbf{z}}}\left[-\log{\left(\frac{\exp{(\boldsymbol{\mathbf{z}}\cdot\boldsymbol{\mathbf{z}}^{+})}}{\exp{(\boldsymbol{\mathbf{z}}\cdot\boldsymbol{\mathbf{z}}^{+})}+\eta\exp{(\boldsymbol{\mathbf{z}}\cdot\boldsymbol{\mathbf{z}}^{-})}}\right)}\right] (2) Here, \eta serves as a hardness parameter of the negative samples. The overall objective function \mathcal{L}_{\CL} for CLNTM is derived as: \mathcal{L}_{\CL}=\mathcal{L}+\mathcal{L}_{\operatorname{InfoNCE}} . In this equation, \mathcal{L} represents the VAE loss function described in Eq. ( 1 ). In their implementation 2 2 2 https://github.com/nguyentthong/CLNTM , the authors set \eta=0.5 . Fig. 3: A general framework for the VAE-based topic models that uses negative sampling methodology on the encoder. III-C VAE-DNS NQTM [ 20 ] and CombinedTM-Neg [ 12 ] are VAE-based topic models incorporating negative sampling in the decoder. A general framework for VAE-DNS is shown in Figure 4 . In NQTM, the goal is diverse topic generation for short texts. The process involves excluding top t topics from the document-topic distribution and randomly drawing a negative sample, \boldsymbol{\mathbf{z}}_{\negt} , from the remaining topics. A negative document, \boldsymbol{\mathbf{x}}_{\negt} , is then generated from the top M words of the distribution of the negative topic. This heuristic introduces a positive bias, aiding topic discovery, while the negative samples enhance learning signals. The objective function, \mathcal{L}_{\CL} , combines the VAE loss ( \mathcal{L} ), negative sampling loss ( \mathcal{L}_{\negt} ), and a proposed quantization loss ( \mathcal{L}_{\operatorname{quant}} ) to address topic collapsing in short texts as follows: \mathcal{L}_{\CL}=\mathcal{L}+\mathcal{L}_{\negt}+\mathcal{L}_{\operatorname{quant}} . The CombinedTM-Neg extends the contextualized topic model by incorporating a negative sampling technique within its decoder. This technique was proposed by us in [ 12 ] . For each input document, a topic vector \boldsymbol{\mathbf{\theta}} is sampled, and a perturbed version \tilde{\boldsymbol{\mathbf{\theta}}}_{\negt} is generated from this vector. This perturbation involves setting the entries corresponding to the top M topics to zero. Subsequently, \tilde{\boldsymbol{\mathbf{\theta}}}_{\negt} is normalized to obtain a probability vector \boldsymbol{\mathbf{\theta}}_{\negt} . Thus, \displaystyle\boldsymbol{\mathbf{\theta}}_{\negt} \displaystyle=\frac{\tilde{\boldsymbol{\mathbf{\theta}}}_{\negt}}{\sum_{i=1}^{T}\tilde{{\theta}}_{\negt}[i]} \displaystyle\text{where }\tilde{{\theta}}_{\negt}[i] \displaystyle=\begin{cases}0&\text{if topic $i$ is included in top $M$ topics,}\\ {\theta}[i]&\text{otherwise}\end{cases} (3) It is assumed that M