Benchmarking Dimensionality Reduction Techniques for Spatial Transcriptomics

Md Ishtyaq Mahmud 1 , Veena Kochat 3 , Suresh Satpati 3 , Jagan Mohan Reddy Dwarampudi 1 , Kunal Rai 3 , Tania Banerjee 1,2 · 2025

Read Benchmarking Dimensionality Reduction Techniques for Spatial Transcriptomics by Md Ishtyaq Mahmud 1 , Veena Kochat 3 , Suresh Satpati 3 , Jagan Mohan Reddy Dwarampudi 1 , Kunal Rai 3 , Tania Banerjee 1,2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-13344.txt.

Introduction

Md Ishtyaq Mahmud, Veena Kochat and four co-authors benchmark shrink methods in 2025 for spatial transcriptomics beyond PCA. Six methods span linear, factorization, and neural paths. A Xenium cancer set drives the tests with varied latent sizes and cluster scales.

Scores cover rebuild error, variance kept, cluster cohesion, and two bio marks for marker fit and exclusion. PCA is the fast base. Factorization tops marker enrichment. Variational nets balance rebuild and clarity. Pareto picks guide settings. Exclusion-guided reassignment lifts marker fit across all methods. Choice should follow analysis goals, not habit.

Catalog record

CREATORMd Ishtyaq Mahmud 1 , Veena Kochat 3 , Suresh Satpati 3 , Jagan Mohan Reddy Dwarampudi 1 , Kunal Rai 3 , Tania Banerjee 1,2
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-13344

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Benchmarking Dimensionality Reduction Techniques for Spatial Transcriptomics Thanks: 1 Department of Electrical and Computer Engineering, University of Houston, Houston, TX, USA Thanks: 2 Department of Information Science Technology, University of Houston, Sugar Land, TX, USA Thanks: 3 Department of Genomic Medicine and MDACC Epigenomics Therapy Initiative (METI), MD Anderson Cancer Center, Houston, TX, USA Thanks: Emails: [email protected], [email protected] Md Ishtyaq Mahmud 1 , Veena Kochat 3 , Suresh Satpati 3 , Jagan Mohan Reddy Dwarampudi 1 , Kunal Rai 3 , Tania Banerjee 1,2 Abstract . We introduce a unified framework for evaluating dimensionality reduction techniques in spatial transcriptomics beyond standard PCA approaches. We benchmark six methods—PCA, NMF, autoencoder, VAE, and two hybrid embeddings—on a cholangiocarcinoma Xenium dataset, systematically varying latent dimensions ( k k =5-40) and clustering resolutions ( ρ \rho =0.1-1.2). Each configuration is evaluated using complementary metrics including reconstruction error, explained variance, cluster cohesion, and two novel biologically-motivated measures: Cluster Marker Coherence (CMC) and Marker Exclusion Rate (MER). Our results demonstrate distinct performance profiles: PCA provides a fast baseline, NMF maximizes marker enrichment, VAE balances reconstruction and interpretability, while autoencoders occupy a middle ground. We provide systematic hyperparameter selection using Pareto optimal analysis and demonstrate how MER-guided reassignment improves biological fidelity across all methods, with CMC scores improving by up to 12% on average. This framework enables principled selection of dimensionality reduction methods tailored to specific spatial transcriptomics analyses. 1. Introduction Dimensionality reduction is a cornerstone of modern spatial transcriptomics pipelines, transforming each cell’s high-dimensional gene-expression profile into a compact embedding that both denoises the data and highlights biologically meaningful variation. In practice, the de facto standard for dimensionaliy reduction is Principal Component Analysis (PCA), a linear projection that maximizes variance along orthogonal axes. However, PCA may miss nonlinear or parts-based structure. In this work, we systematically benchmark two additional classes of dimensionality methods (shown in Figure 1 : • Linear methods : PCA and Non-negative Matrix Factorization (NMF). NMF constrains both factors and loadings to be nonnegative, yielding additive, parts-based representations. • Deep nonlinear methods : Autoencoders (AE) and Variational Autoencoders (VAE), which learn flexible encoder–decoder networks that can capture complex manifolds in gene-express-ion space. • Hybrid methods : Concatenated embeddings, namely PCA + NMF and VAE + NMF, that combine complementary linear and nonlinear features. To evaluate how each embedding shapes downstream clustering, we employ a suite of metrics: • Reconstruction fidelity: Mean squared error (MSE) and explained variance of the embedding. • Clustering quality: Silhouette score and Davies–Bouldin Index (DBI). • Biological coherence: two novel metrics — Cluster Marker Coherence (CMC), the fraction of cells in each cluster expressing its marker genes, and Marker Exclusion Rate (MER), the fraction of cells that would express another cluster’s markers more strongly. • Gene-set enrichment: average enrichment of known marker-gene sets per cluster. Because clustering solely on low-dimensional embeddings can misassign cells whose original transcriptomes express another cluster’s markers, we introduce a lightweight post-processing step. Any cell with higher aggregate marker expression in a different cluster is reassigned accordingly, guided by MER. We then perform a before-versus-after analysis to quantify how this refinement improves marker coherence and downstream clustering metrics. Note that we exclude spatially-augmented algorithms from this analysis, reserving them for future work. Our contributions are threefold: (1) To our knowledge, this represents the first systematic comparison of PCA, NMF, AE, VAE, and hybrid embeddings applied to the same spatial transcriptomics data. By controlling for dataset variability, we can directly assess how mathematical differences in dimensionality reduction translate into biological differences in the resulting cluster assignments and marker gene expression patterns. (2) We introduce two novel clustering metrics, Cluster Marker Coherence (CMC) and Marker Expression Ratio (MER), that are interpretable and biologically motivated, and designed to assess how well clustering results align with marker gene expression patterns. (3) We present a simple yet novel MER-guided cell reassignment algorithm that substantially enhances marker coherence through post-processing refinement of initial clustering results. The remainder of the paper is organized as follows. Section 2 reviews related work. Section 3 presents a unified mathematical framework for dimensionality reduction methods. Section 4 describes our datasets, preprocessing, implementation of the dimensionality reduction methods and evaluation metrics. Section 5 reports our experimental results and discusses practical implications, and finally, Section 6 concludes the paper. Figure 1 . An overview of the dimensionality reduction methods evaluated in this work, grouped into linear (PCA, NMF), non-linear (AE, VAE), and hybrid (PCA+NMF, VAE+NMF) approaches. The hybrid approach uses concatenated embeddings from two methods. 2. Related Work Dimensionality reduction is a fundamental step in the analysis of high-dimensional single-cell and spatial transcriptomics data, underpinning downstream tasks such as clustering, visualization, and tissue-domain discovery. While many dimensionality reduction methods exist, few studies have systematically compared their performance and interpretability in a spatial-omics setting. Classical and Matrix-Factorization Methods: PCA remains the de facto choice in single-cell pipelines (e.g., Seurat ( Satija et al. 2015 ) , Scanpy ( Wolf et al. 2018 ) ) due to its analytical tractability and speed. However, its linear projections can miss nonlinear biological variation or spatial context. NMF uncovers additive, parts-based gene programs ( Brunet et al. 2004 ) and has been applied to spatial data for interpretable domain discovery ( Townes and Engelhardt 2023 ) . Its nonnegativity constraint yields intuitive gene signatures, but it cannot model nonlinear interactions. Deep Learning Approaches: AEs and VAEs provide flexible, nonlinear mappings that can capture complex manifolds in single-cell expression ( Eraslan et al. 2019 ; Lopez et al. 2018 ) . Despite their promise, these models have rarely been benchmarked in spatial transcriptomics, and challenges remain around implementation, overfitting, interpretability, and alignment with spatial structure. Benchmarking Studies Several recent studies have compared dimensionality reduction techniques on single-cell RNA-seq data, evaluating clustering tightness, global structure preservation, and visualization clarity ( Duò et al. 2020 ; Sun et al. 2024 ) , including surveys of existing techniques such as PCA, t-SNE, and scVI ( Lopez et al. 2018 ) . However, these analyses typically do not provide in-depth comparisons by applying multiple techniques to identical datasets with systematic parameter exploration. Our work fills this gap by providing a unified mathematical framework alongside comprehensive empirical evaluation of linear (PCA, NMF), nonlinear (AE, VAE), and hybrid (PCA+NMF, VAE+NMF) approaches on the same spatial transcriptomics dataset. We explicitly measure embedding quality, introduce novel metrics for marker coherence (CMC) and marker exclusion rate (MER), and analyze how different dimensionality reduction methods affect cluster composition and biological interpretation within tissue microarray samples. 3. Theoretical Framework and Interpretability In this section we lay out a unified view of the four dimensionality-reduction techniques under study, PCA, NMF, AE and VAE, in terms of their core optimization objectives, and briefly discuss how these formulations influence interpretability in the spatial-transcriptomics setting: Let X\in\mathbb{R}^{n\times d} be our normalized cell–by–gene expression matrix (with n cells and d genes), and let Z\in\mathbb{R}^{n\times k} denote its low-dimensional embedding ( k\ll d ). We then have: (1) Principal Component Analysis (PCA): PCA solves the minimization problem \min_{Z,W}\|X-ZW^{\top}\|_{F}^{2}\quad\text{subject to }W^{\top}W=I, where W\in\mathbb{R}^{d\times k} is an orthonormal basis and Z=XW is the projection of X onto the k -dimensional subspace ( Jolliffe and Cadima 2016 ; Zhao et al. 2021 ) .

Read the complete text