S2WTM: Spherical Sliced-Wasserstein Autoencoder for Topic Modeling

Suman Adhya, Debarshi Kumar Sanyal · 2025

Read S2WTM: Spherical Sliced-Wasserstein Autoencoder for Topic Modeling by Suman Adhya, Debarshi Kumar Sanyal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-12451.txt.

Introduction

Suman Adhya and Debarshi Kumar Sanyal propose a topic model that places document and topic representations on a sphere, where direction captures meaning. Sphere-based models suit high-dimensional text, but the standard variational approach often suffers posterior collapse, where the latent code stops carrying information. Their model aims to keep the spherical structure without that failure.

Instead of the usual divergence penalty, they align the gathered posterior with a spherical prior using a sliced Wasserstein distance adapted to the sphere. In experiments, the model produced topics that were more coherent and more varied than leading rivals, and it improved results on downstream tasks that use the learned representations.

Catalog record

CREATORSuman Adhya, Debarshi Kumar Sanyal
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-12451

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

S2WTM: Spherical Sliced-Wasserstein Autoencoder for Topic Modeling Suman Adhya, Debarshi Kumar Sanyal Abstract Modeling latent representations in a hyperspherical space has proven effective for capturing directional similarities in high-dimensional text data, benefiting topic modeling. Variational autoencoder-based neural topic models (VAE-NTMs) commonly adopt the von Mises-Fisher prior to encode hyperspherical structure. However, VAE-NTMs often suffer from posterior collapse, where the KL divergence term in the objective function highly diminishes, leading to ineffective latent representations. To mitigate this issue while modeling hyperspherical structure in the latent space, we propose the Spherical Sliced Wasserstein Autoencoder for Topic Modeling (S2WTM). S2WTM employs a prior distribution supported on the unit hypersphere and leverages the Spherical Sliced-Wasserstein distance to align the aggregated posterior distribution with the prior. Experimental results demonstrate that S2WTM outperforms state-of-the-art topic models, generating more coherent and diverse topics while improving performance on downstream tasks. github.com/AdhyaSuman/S2WTM 1 Introduction Topic modeling aims to uncover latent topics in a document corpus by representing each document as a probabilistic mixture of topics. Latent Dirichlet Allocation (LDA) Blei et al. 2003 and Probabilistic Latent Semantic Indexing (pLSI) Hofmann 1999 typically assume a multinomial distribution over words. However, this assumption treats words as independent, ignoring important syntactic and semantic relationships. Moreover, classical topic models rely on computationally expensive inference techniques, such as Gibbs sampling and variational inference. To address these limitations, Variational Autoencoder-based Neural Topic Models (VAE-NTMs) were introduced by Miao et al. 2016 , offering a more scalable and flexible alternative. VAE-NTMs typically use a Gaussian prior for the latent space. However, in high-dimensional settings, Gaussian distributions exhibit the “soap bubble effect" Davidson et al. 2018 , where probability mass concentrates near the surface of a hypersphere rather than around the mean. This effect undermines the utility of Euclidean distances, which tend to become uniform and less informative as dimensionality increases—a core aspect of the curse of dimensionality Aggarwal et al. 2001 . In contrast, cosine similarity, which measures the angular difference between vectors, remains a more robust metric in such regimes Dhillon and Sra 2003 . These observations motivate the use of spherical latent space modeling, where data is embedded on the surface of a hypersphere and similarities are measured via angles rather than distances. This is particularly effective in large-scale applications involving directional data, such as text categorization and gene expression analysis Dhillon and Sra 2003 ; Banerjee et al. 2005 , where the orientation of data vectors carries meaningful information. The von Mises-Fisher (vMF) distribution naturally models such data and has demonstrated superior performance over models with Gaussian priors in clustering and topic modeling tasks Reisinger et al. 2010 ; Li et al. 2016 ; Batmanghelich et al. 2016 . Recent advancements in topic modeling, such as vONTSS Xu et al. 2023 , have leveraged the \mathcal{S} -VAE framework and integrated supervised signals via optimal transport to enhance topic coherence and classification performance. Despite these advancements, VAE-based models often suffer from posterior collapse Bowman et al. 2016 , wherein the latent representations fail to encode meaningful information. This issue occurs when the KL divergence term in the loss function approaches zero, causing the posterior distribution to converge to the uninformative prior. A detailed analysis of this phenomenon, including its formal definition and decomposition via ELBO surgery, is provided in Appendix A . To address these challenges, we propose S2WTM (Spherical Sliced-Wasserstein Autoencoder for Topic Modeling), an autoencoder-based topic model that models the latent space on the unit hypersphere. S2WTM introduces a flexible prior design by treating the choice of prior distribution as a hyperparameter, supporting three alternatives: the vMF distribution, a Mixture of vMF (MvMF) distributions, and the uniform distribution on the hypersphere. To align the aggregated posterior with the chosen prior while preserving spherical geometry, S2WTM employs the Spherical Sliced-Wasserstein (SSW) distance, effectively mitigating the posterior collapse problem. Contributions: In summary, our work presents the following key contributions: • We introduce S2WTM, a neural topic model leveraging the Spherical Sliced-Wasserstein autoencoder framework to align the aggregated posterior with a hyperspherical prior, ensuring effective latent space modeling. • We explore three distinct choices of prior distributions defined on the unit hypersphere for modeling the latent space. • We evaluate the performance of S2WTM using quantitative metrics, qualitative analysis, task-specific performance, and LLM-based human-comparable evaluations, demonstrating its advantages over state-of-the-art models. • We empirically validate the benefits of moving beyond the conventional Euclidean latent space, highlighting performance improvements in topic modeling. 2 Preliminaries This section introduces three selected distributions defined on a unit hypersphere. We then discuss optimal transport metrics, focusing on the Wasserstein distance, sliced Wasserstein distance, and spherical sliced-Wasserstein distance. 2.1 Distributions on the Unit Hypersphere vMF distribution: The vMF distribution is defined on the unit (K-1) -sphere \mathcal{S}^{K-1} embedded in \mathbb{R}^{K} . The vMF distribution is parameterized by its mean direction ( \bm{\mathbf{\mu}}\in\mathcal{S}^{K-1} ), a unit vector indicating the central direction of the distribution, and its concentration ( \kappa\in\mathbb{R}^{+}\cup\{0\} ), a non-negative scalar controlling the dispersion around \bm{\mathbf{\mu}} . A larger \kappa indicates a higher concentration. For \bm{\mathbf{x}}\in\mathcal{S}^{K-1} with \kappa\geq 0 the vMF density function is: \vmf(\bm{\mathbf{x}};\bm{\mathbf{\mu}},\kappa)=c_{p}(\kappa)\exp{(\kappa\bm{\mathbf{\mu}}^{\intercal}\bm{\mathbf{x}})} where c_{K}(\kappa) is the normalization constant: c_{K}(\kappa)=\frac{\kappa^{\frac{K}{2}-1}}{(2\pi)^{\frac{K}{2}}I_{\frac{K}{2}-1}(\kappa)} with I_{v} denotes the modified Bessel function of the first kind of order v . The detailed sampling from the vMF distribution Ulrich 1984 is described in Appendix B . MvMF distribution: The MvMF distribution is a convex combination of vMF distributions. If the mixture contains T vMF components, then the density is defined for all points \bm{\mathbf{x}}\in\mathcal{S}^{K-1} as \mvmf(\bm{\mathbf{x}};\bm{\mathbf{\Psi}})=\sum_{t=1}^{T}\alpha_{t}f_{t}(\bm{\mathbf{x}}|\bm{\mathbf{\psi}}_{t}) . Here, each component f_{t} is a vMF distribution with parameters \bm{\mathbf{\psi}}_{t}=(\bm{\mathbf{\mu}}_{t},\kappa_{t}) , and \bm{\mathbf{\Psi}} represents the set of all such parameters. The mixture proportions (\alpha_{t})_{1\leq t\leq T} satisfy \alpha_{t}\geq 0 and \sum_{t=1}^{T}\alpha_{t}=1 . The sampling technique from MvMF requires first selecting a component i\sim\operatorname{Cat}(\alpha_{1},\alpha_{2},...,\alpha_{T}) , then generating a sample from the chosen vMF distribution. Uniform distribution on the unit hypersphere: The uniform distribution on \mathcal{S}^{K-1} assigns equal probability density to all directions. Sampling is performed by drawing from a standard Gaussian in \mathbb{R}^{K} and applying \ell_{2} normalization. 2.2 Spherical Sliced-Wasserstein Distance: The Spherical Sliced-Wasserstein (SSW) distance is a variant of the Sliced-Wasserstein (SW) distance, a computationally efficient approximation of the Wasserstein distance. SSW adapts SW for distributions on the unit hypersphere or other spherical domains. We first review the Wasserstein and SW distances. Definition 1 (Wasserstein Distance Villani et al. 2009 ; Figalli and Villani 2011 ) . Let p\geq 1 . The p -Wasserstein distance between \mu,\nu\in\mathcal{P}_{p}(\mathbb{R}^{d}) is W_{p}^{p}(\mu,\nu)=\inf_{\gamma\in\Gamma(\mu,\nu)}\ \int_{\mathbb{R}^{d}\times\mathbb{R}^{d}}d(x,y)^{p}\ \mathrm{d}\gamma(x,y) (1) where \mathcal{P}_{p}(\mathbb{R}^{d}) is the set of Borel probability measures on \mathbb{R}^{d} with finite p -moments, and \Gamma(\mu,\nu) are joint measures with marginals \mu and \nu . Computing W_{p} is computationally expensive, scaling as \mathcal{O}(n^{3}\log n) for discrete measures with n samples Peyré et al. 2019 . However, W_{p} has a closed form for univariate distributions: W_{p}^{p}(\mu,\nu)=\int_{0}^{1}|F_{\mu}^{-1}(t)-F_{\nu}^{-1}(t)|^{p}dt, (2) where F_{\mu}^{-1} and F_{\nu}^{-1} are the quantile functions. This leads to the SW, exploiting the efficiency of one-dimensional optimal transport. Definition 2 (Sliced Wasserstein Distance Julien et al. 2011 ; Bonneel et al. 2015 ) . For p\geq 1 , the p -SW between \mu,\nu\in\mathcal{P}_{p}(\mathbb{R}^{d}) is SW_{p}^{p}(\mu,\nu)=\int_{S^{d-1}}W_{p}^{p}\Big((\mathcal{R}\mu)^{\theta},(\mathcal{R}\nu)^{\theta}\Big)d\theta (3) where (\mathcal{R}\eta)^{\theta} is the Radon transform, projecting \eta onto the line spanned by \theta\in\mathcal{S}^{d-1} . SW reduces computation to

Read the complete text