Graph Topic Modeling for Documents with Spatial or Covariate Dependencies
Yeo Jin Jung · 2024
Read Graph Topic Modeling for Documents with Spatial or Covariate Dependencies by Yeo Jin Jung (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14477.txt.
Introduction
Yeo Jin Jung and Claire Donnat present, in a 2024 paper, a topic model that uses what is known about documents to sort their themes better. Topic models estimate the mix of themes in each document. Facts about a document, such as its place or source, and likeness between documents, carry clues that standard models ignore. Bayesian fixes for this are slow and lack firm guarantees.
Their method recasts documents as nodes in a graph, with edges for likeness. A fast, graph smoothed factorization then pushes similar documents toward similar theme mixes. The authors bound the method's error with high probability, and design a special cross check to set its tuning values. On synthetic data and three real corpora, it beat Bayesian rivals on accuracy and speed. That makes it practical for large collections with rich side data.
Catalog record
| CREATOR | Yeo Jin Jung |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 144 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-14477
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Graph Topic Modeling for Documents with Spatial or Covariate Dependencies Yeo Jin Jung Abstract We address the challenge of incorporating document-level metadata into topic modeling to improve topic mixture estimation. To overcome the computational complexity and lack of theoretical guarantees in existing Bayesian methods, we extend probabilistic latent semantic indexing (pLSI), a frequentist framework for topic modeling, by incorporating document-level covariates or known similarities between documents through a graph formalism. Modeling documents as nodes and edges denoting similarities, we propose a new estimator based on a fast graph-regularized iterative singular value decomposition (SVD) that encourages similar documents to share similar topic mixture proportions. We characterize the estimation error of our proposed method by deriving high-probability bounds and develop a specialized cross-validation method to optimize our regularization parameters. We validate our model through comprehensive experiments on synthetic datasets and three real-world corpora, demonstrating improved performance and faster inference compared to existing Bayesian methods. Keywords: graph cross validation, graph regularization, latent dirichlet allocation, total variation penalty 1 Introduction Consider a corpus of n documents, each composed of words (or more generally, terms) from a vocabulary of size p . This corpus can be represented by a document-term matrix D\in\mathbb{N}^{n\times p} , where each entry D_{ij} denotes the number of times term j appears in document i . The objective of topic modeling is to retrieve low-dimensional representations of the data by representing each document as a mixture of latent topics, defined as distributions over term frequencies. In this setting, each document D_{i\cdot}\in\mathbb{R}^{p} is usually assumed to be sampled from a multinomial distribution with an associated probability vector M_{i\cdot}\in\mathbb{R}^{p} that can be decomposed as a mixture of K topics. In other words, letting N_{i} denote the length of document i : \forall i=1,\cdots,n,\qquad D_{i\cdot}\sim\text{Multinomial}(N_{i},M_{i\cdot}),\qquad\text{with }M_{i\cdot}=\sum_{k=1}^{K}W_{ik}A_{k\cdot} (1) In the previous equation, W_{ik} corresponds to the proportion of topic k in document i , and the vector W_{i\cdot} provides a low-dimensional representation of document i in terms of its topic composition. Each entry A_{kj} of the vector A_{k\cdot}\in\mathbb{R}^{p} corresponds to the expected frequency of word j in topic k . Since document lengths \{N_{i}\}_{i=1}^{n} are usually treated as nuisance variables, most topic modeling approaches work in fact directly with the word frequency matrix X=\text{diag}(\{\frac{1}{N_{i}}\}_{i=1\cdots n})D , which can be written in a “signal + noise” form as: X=M+Z=WA+Z. (2) Here, M\in\mathbb{R}^{n\times p} is the true signal whose entry M_{ij} denotes the expected frequency of word i in document j , while Z=X-M denotes some centered multinomial noise. The objective of topic modeling is thus to estimate W and A from X . Originally developed to reduce document representations to low-dimensional latent semantic spaces, topic modeling has been successfully deployed for the analysis of count data in a number of applications, ranging from image processing ( Tu et al. 2016 , Zheng et al. 2015 ) and image annotation ( Feng & Lapata 2010 , Shao et al. 2009 ) , to biochemistry ( Reder et al. 2021 ) , genetics ( Dey et al. 2017 , Kho et al. 2017 , Liu et al. 2016 , Yang et al. 2019 ) , and microbiome studies ( Sankaran & Holmes 2019 , Symul et al. 2023 ) . A notable extension of topic modeling occurs when additional document-level data is available. Although original topic modeling approaches rely solely on the analysis of the empirical frequency matrix X , this additional information has the potential to significantly improve the estimation of the word-topic matrix A and the document-topic mixture matrix W , particularly in difficult inference settings, such as when the number of words per document is small. Examples include: 1. Analyzing tumor microenvironments: In this context, slices of tumor samples are partitioned into smaller regions known as tumor microenvironments, where the frequency of specific immune cell types is recorded ( Chen et al. 2020 ) . Here, documents correspond to microenvironments, and words to cell types. The objective is to identify communities of co-abundant cells (topics), taken here as proxies for tumor-immune cell interactions and potential predictors of patient outcomes. In this setting, we assume that neighboring microenvironments share similar topic proportions. Since these microenvironments are inherently small, leveraging the spatial smoothness of the mixture matrix W can significantly improve inference ( Chen et al. 2020 ) . We develop this example in further detail in Section 4.1 . 2. Microbiome studies: Topic models have also been proven to be extremely useful in microbiome analysis ( Sankaran & Holmes 2019 , Symul et al. 2023 ) . In this setting, the data consists of a microbiome count matrix recording the amount of different types of bacteria found in each sample. In this case, microbiome samples are identified to documents, with bacteria playing the roles of the vocabulary, and the goal becomes to identify communities of co-abundant bacteria ( Sankaran & Holmes 2019 ) . When additional covariate information is available (such as age, gender, and other demographic details), we can expect similar samples (documents) to exhibit similar community compositions (topic proportions). 3. The analysis of short documents , such as a collection of scientific abstracts or recipes: In this case, while recipes might be short, information on the origin of the recipe can help determine the topics and mixture matrix more accurately by leveraging the assumption that recipes of neighboring countries will typically share similar topic proportions. We elaborate on this example in greater detail in Section 4.3 . Prior works. Previous attempts to incorporate metadata in topic estimation have focused on the Bayesian extensions of latent Dirichlet allocation (LDA) model of Blei et al. (2001) . By and large, these methods typically incorporate the metadata—often in the form of a covariate matrix—within a prior distribution ( Blei & Lafferty 2006a , Blei & Lafferty 2006b , Roberts et al. 2014 , Mcauliffe & Blei 2007 ) . However, these models (a) are difficult to adapt to different types of covariates or information encoded as a graph, and (b) typically lack theoretical guarantees. Recent work by Chen et al. (2020) proposes extending LDA to analyze documents with known similarities by smoothing the topic proportion hyperparameters along the edges of the graph. However, this method does not empirically yield spatially smooth structures (see Sections 3.4 and 4 ), and significantly increases the algorithm’s running time. In the frequentist realm, probabilistic latent semantic indexing (pLSI) has gained renewed interest over the past five years. Similar to LDA, it effectively models documents as bags of words but differs by treating matrices A and W as fixed parameters. In particular, new work by Ke & Wang (2017) and Klopp et al. (2021) suggest procedures to reliably estimate the topic matrix A and mixture matrix W , characterizing consistency and efficiency through high-probability error bounds Although recent work has begun investigating the use of structures in pLSI-based topic models, most approaches have limited this to considering various versions of sparsity ( Bing et al. 2020 , Wu et al. 2023 ) or weak sparsity ( Tran et al. 2023 ) for the topic matrix A . To the best of our knowledge, no pLSI approach has yet been proposed that effectively leverages similarities between documents nor characterizes the consistency of these