Robust subspace structure discovery for cell type identification in scRNA-seq data Authors: Xianyong Zhou, Xindian Wei, Cheng Liu, Wenjun Shen, Ping Xuan, Si Wu, Hau-San Wong Published: 2025 ABSTRACT AbstractSingle-cell RNA sequencing (scRNA-seq) technology has transformed gene expression studies by enabling analysis at the individual cell level, offering unprecedented insights into cellular heterogeneity. A key challenge in scRNA-seq data analysis is cell type identification, which requires grouping cells with similar gene expression profiles using unsupervised clustering methods. However, the high dimensionality, inherent noise, and significant sparsity of scRNA-seq data present substantial obstacles to accurately determining relationships among cell samples. To address these challenges, we propose a novel deep subspace clustering approach for cell type identification that captures a more reliable subspace structure from scRNA-seq data. Our method leverages a robust self-representation learning framework to effectively characterize and learn the underlying cluster structure. This framework is optimized through an integrated strategy combining a structure-guided approach with an optimal transport algorithm, enhancing the robustness of the subspace clustering process. By mitigating the effects of noise and sparsity in scRNA-seq data, this approach enables more accurate cell clustering. Experimental results on 18 real scRNA-seq datasets demonstrate that our method outperforms several state-of-the-art clustering approaches tailored for scRNA-seq data, excelling in both accuracy and interpretability.Keywords: Deep subspace clustering, ScRNA-seq data analysis, Cell type identification Introduction The development of single-cell RNA sequencing (scRNA-seq) technology has enabled study gene expression analysis at the single-cell level. This advanced technology provides new insights into biological mechanisms, such as tumor microenvironment analysis and personalized medicine [1]. A critical task in scRNA-seq data analysis is cell type identification, which involves grouping cells based on similar gene expression patterns. This can be effectively achieved through unsupervised clustering [2–6]. Several clustering methods have been proposed to identify cell types from scRNA-seq profiles [6–8]. A common approach involves a sequential process: first, dimensionality reduction of the gene expression matrix, followed by the application of traditional clustering methods such as k-means , hierarchical clustering and based on graphs. For instance, pcaReduce [9] combines principal component analysis (PCA) with hierarchical clustering clustering, producing cell identification results. Seurat [10] constructs a shared nearest neighbor (SNN) graph based on the principal components of the gene expression matrix to capture the similarity between cells and performs clustering using the Louvain algorithm [11]. Similarly, CIDR [12] addresses missing data through imputation before applying dimensionality reduction and k-means clustering for cell identification. In addition, some methods further enhance clustering performance by using ensemble strategies. For example, SC3 [13] combines results from multiple clustering algorithms to achieve more stable and consistent outcomes. Meanwhile, SIMLR [14] employs multiple kernel functions to capture diverse features and relationships within the data, leading to more accurate cell type identification through clustering. Recently, leveraging the feature representation learning capabilities of deep learning, several deep clustering methods have been successfully applied to scRNA-seq data for cell identification [15–27]. For example, scAce [20] constructs a variational autoencoder to simultaneously learn cell embeddings and cluster assignments while developing an adaptive cluster merging approach, improving clustering results without needing to estimate the number of clusters in advance. SLNMF [21] employs a non-negative matrix factorization algorithm based on network structure learning to identify cell types in scRNA-seq data, transforming the task into a constrained optimization problem. Another example, ScCAEs [22], introduces a deep embedding clustering method based on a convolutional autoencoder and soft k-means clustering, addressing high dropout rates and noise by simultaneously learning feature representations and performing clustering. In addition to traditional embedding learning, several methods incorporate the structural relationships between cells using graph neural networks (GNNs) to capture more reliable representations for cell clustering. For instance, scDSC [23] combines a ZINB-based autoencoder, a graph neural network module, and a mutual-supervision module to effectively learn and cluster scRNA-seq data, capturing both the intrinsic properties of individual cells and their relationships. Similarly, scGAC [24] constructs a cell map, refines it through network denoising, learns a cluster-friendly representation using a graph attention autoencoder, and finally clusters cells using a self-optimization method. scGNN [25] integrates a graph convolutional network (GCN) into a multi-autoencoder framework to improve cell clustering. It constructs a cell graph using a feature autoencoder, refines cell representations through a GCN-based autoencoder, and applies k-means clustering, with iterative refinement of cell representations by cluster-specific autoencoders. Accurately estimating cell relationships and obtaining robust feature representations are central challenges in cell clustering for cell type identification. Although several studies have demonstrated effectiveness in clustering tasks, most existing methods still face critical challenges due to the high dropout rates and noise inherent in scRNA-seq data. These intrinsic properties often result in unstable clustering outcomes, making cell type identification less reliable. To overcome these challenges, we introduce a novel deep subspace clustering framework specifically designed for cell type identification. Our method first employs a structure-guided learning strategy that mitigates the tendency of deep subspace clustering models to converge to suboptimal solutions, particularly when initialized poorly or when confronted with high levels of noise in the data. Additionally, we incorporate the Wasserstein distance into the self-representation framework, where it serves as a regularization term to enhance the quality of the self-representation. This self-representation is crucial for capturing relationships among samples and for effective clustering, ultimately guiding the overall model learning process. The Wasserstein-regularized self-representation learning is efficiently optimized using a standard optimal transport algorithm. Overall, by combining a structure-guided learning strategy with optimal transport-based regularization, our method significantly strengthens the subspace clustering framework, rendering it more robust to the challenges inherent in scRNA-seq data. The overview of the proposed can be found in Fig. 1. Overview of the proposed method. The framework first encodes each cell expression into a latent space via a deep autoencoder (deep subspace representation). A initial graph guides self-representation learning to improve robustness against noise. The Wasserstein self-representation step refines the affinity matrix using optimal transport to handle sparsity and dropout. Parameters are updated through alternating optimization, and final cell type assignments are obtained via spectral clustering The main contributions of this work are summarized as follows: We propose a novel deep subspace clustering method for cell clustering in scRNA-seq data. By uncovering robust cluster structures, our approach accurately captures cell similarities and yields more precise clustering outcomes. We propose a novel deep subspace clustering method for cell clustering in scRNA-seq data. By uncovering robust cluster structures, our approach accurately captures cell similarities and yields more precise clustering outcomes. Our method enhances clustering robustness through two key aspects: a structure-guided learning strategy that mitigates convergence to suboptimal solutions under high noise and dropout conditions, and a Wasserstein-regularized self-representation framework that refines the inter-cell relationship estimation. Our method enhances clustering robustness through two key aspects: a structure-guided learning strategy that mitigates convergence to suboptimal solutions under high noise and dropout conditions, and a Wasserstein-regularized self-representation framework that refines the inter-cell relationship estimation. Extensive experiments on 18 real-world scRNA-seq datasets, benchmarked against 12 state-of-the-art methods, demonstrate that our approach not only outperforms existing techniques in clustering accuracy but also provides biologically meaningful insights by validating cell type annotations and differential expression analysis. Extensive experiments on 18 real-world scRNA-seq datasets, benchmarked against 12 state-of-the-art methods, demonstrate that our approach not only outperforms existing techniques in clustering accuracy but also provides biologically meaningful insights by validating cell type annotations and differential expression analysis. Method In this section, we provide a detailed explanation of the proposed algorithm. We start by introducing the foundational concepts of deep subspace clustering [28], followed by the introduction of two key innovations in the proposed method: a structure-guided learning strategy and Wasserstein-regularized self-representation. These components are seamlessly integrated into a unified optimization framework to improve the robustness and accuracy of cell type identification. Deep subspace clustering Let represent an scRNA-seq dataset, where D denotes the number of genes and N the number of cells. For subspace clustering, assuming the data points of are drawn from K distinct multilinear subspaces, the central concept is self-expressiveness: each sample can be expressed as a linear combination of others. Mathematically, this is represented as , where is the self-expression coefficient matrix. Ideally, if and lie in the same subspace, and otherwise. This encourages to exhibit a block diagonal structure, where the number of blocks corresponds to the number of subspaces. This structure can be enforced by minimizing appropriate norms of , based on the assumption of independent subspaces, which can be formulated as follows: In deep subspace clustering, autoencoder networks can be employed as unsupervised models to project the original features into a lower-dimensional latent representation space. Specifically, the encoder network is used to learn the latent representation of the raw data: where represents the encoder function, and denotes the encoder parameters. The corresponding decoder network is then utilized to reconstruct the original data from the latent representation: where represents the decoder network, and are the decoder parameters. The reconstruction loss, , is employed to learn a more discriminative representation of the data by quantifying the difference between the input data and the reconstructed data. This loss is defined as follows: In addition, with the latent representation learned through the autoencoder network, the clustering module with self-representation loss for obtaining the self-representation matrix in Eq. (1) can be reformulated as follows: where the first term measures the reconstruction error of the latent representation using the self-representation matrix , and the second term is a regularization term that controls the complexity of . The regularization parameter balances the trade-off between the reconstruction accuracy and the sparsity of . Structure guided learning strategy A common pitfall in deep subspace clustering is its tendency to converge to suboptimal solutions, particularly when initialization is poor or the data is noisy. In scRNA-seq datasets, these issues are amplified by high dropout rates and inherent biological variability, often leading to unstable clustering results. For this purpose, we first propose a structure-guided learning strategy. This approach leverages the initial geometric structure of the data to guide the self-representation learning process. We begin by constructing a initial graph using a self-representation learning approach from raw features to capture the initial relationships between cells based on their gene expression profiles. This graph captures the initial relationships between cells based on their gene expression profiles, providing a robust representation of the underlying structure. The initial graph then serves as a supervision signal to guide the view-specific self-representation learning process, formalized as: where the Kullback–Leibler (KL) divergence enforces similarity between and the initial graph . This loss function aids in preventing the model from converging to unreliable local minima by anchoring the self-representation to the graph structure. However, employing as a strict constraint throughout training can overly restrict the flexibility of the model. To address this, we adopt as a reliable initialization rather than a permanent supervisory signal. By starting with , the model avoids poor local minima early on and can explore the solution space more effectively. This balance between guidance and flexibility enables the learned representations to more accurately capture the true relationships among cells, thereby improving the overall clustering performance on scRNA-seq data. Wasserstein self-representation learning Although deep subspace clustering effectively reveals the nonlinear information embedded within the original features and produces clustering based on self-representation, the high dropout rates and noise inherent in scRNA-seq data pose significant challenges. These factors often lead to unstable self-representation, which can degrade the clustering performance. To achieve more accurate cell clustering, we introduce an optimal transport strategy to boost the quality of self-representation with Wasserstein metric and better capture relationships among samples. Optimal transport traditionally identifies the transport plan that minimizes the cost or distance between two probability distributions. Definition 1 Given probability distributions x and y, the cost of transporting mass from to is given by a cost matrix and , and the related optimal transport distance can be defined as: For the discrete case, with and representing the probability masses at points and , respectively, the OT distance is defined as: For optimal transport in Eqs. (7) and (8), constructing the cost matrix is essential for optimal transport. In this work, we apply optimal transport to redefine the self-representation learning for , aiming to better capture relationships among samples and guide model learning. For this self-representation learning, we define the cost matrix as , reflecting the relationships among samples in the latent space. The cell–cell distance based self-representation is then computed based on using the optimal transport distance: The resulting optimization can be solved using the Sinkhorn-Knopp algorithm to obtain the optimal transport plan , representing the optimal cell-cell relationship matching matrix. We further reformulate self-representation learning by incorporating this optimal transport plan : As a result, the reformulated Wasserstein self-representation learning incorporates the learned optimal transport plan, allowing the self-representation loss to propagate backward, guiding the entire model learning process more effectively. Overall, the total loss function of the proposed model can be defined as: Optimization We employ an alternating optimization approach to iteratively update the parameters of the deep subspace clustering network and the optimal transport plan for self-representation learning. The training process can be outlined in the following steps: First, we apply a pre-training strategy to train the autoencoder using the reconstruction loss , while simultaneously obtaining an initial graph, which serves as a reliable starting point for self-representation. This provides a reliable initialization for the deep embedding process. First, we apply a pre-training strategy to train the autoencoder using the reconstruction loss , while simultaneously obtaining an initial graph, which serves as a reliable starting point for self-representation. This provides a reliable initialization for the deep embedding process. In subsequent epochs, we alternate between updating the deep subspace clustering parameters and estimating the optimal transport plan . Specifically, the deep subspace clustering model, incorporating Wasserstein self-representation learning (Eq. 10), is optimized using the standard ADMM optimization method. Afterward, the Sinkhorn-Knopp algorithm is used to compute the optimal transport plan , which is then fed back into the self-representation loss (Eq. 9) for refinement. In subsequent epochs, we alternate between updating the deep subspace clustering parameters and estimating the optimal transport plan . Specifically, the deep subspace clustering model, incorporating Wasserstein self-representation learning (Eq. 10), is optimized using the standard ADMM optimization method. Afterward, the Sinkhorn-Knopp algorithm is used to compute the optimal transport plan , which is then fed back into the self-representation loss (Eq. 9) for refinement. Finally, we apply the spectral clustering algorithm to the affinity matrix to obtain the final clustering results for cell type identification. Finally, we apply the spectral clustering algorithm to the affinity matrix to obtain the final clustering results for cell type identification. The optimization process is summarized in Algorithm 1 Optimization algorithm Experiments Experimental setup Datasets and pre-processing In this work, we utilize 18 single-cell datasets from various tissues and platforms to evaluate the effectiveness of the proposed method (detailed dataset information can be found in Table 1). The training data is usually organized as a matrix, where N represents the number of cells and D indicates the gene number. Inspired by [29], for our method, we transform the expression profiles of individual cells into a two-dimensional matrix format, similar to image data, rather than using the original one-dimensional vectors as training samples. This transformation allows the proposed network to better capture non-linear gene-gene dependencies in complex, multi-cell type samples, enhancing its robustness to cases where a small set of genes has very low expression levels. Specifically, the reshaped data is defined as an matrix, where n is the smallest integer such that . If the gene number is smaller than , the remaining entries are padded with zeros. Table 1Details of the 18 evaluated datasetsDatasetDescriptionCellsGenesCell typesPlatformReferencesHuman ESCHuman pluripotent stem cells101818,2947Illumina HiSeq[30]mESCMus musculus28815,9093Illumina Seq[1]mPCTCPancreatic circulating tumor cells11219,3646Genetic Analyzer–Human brainThe adult and fetal human brain cells42021,5179Illumina MiSeq[31]Human pancreasHuman pancreatic islets were obtained from Prodo or NDRI360420,12513Illumina HiSeq[32]Li cellHuman colorectal tumor cells56113,1607Illumina HiSeq[33]Li NMHuman colorectal tumor cells26613,3787Illumina HiSeq[33]LekemiaSingle-cell ATAC-seq39176026Illumina MiSeq[34]SpleenMus musculu cells955214,400510X[35]tcHuman pluripotent stem cells75113,6376Illumina HiSeq[30]ZeiselMouse cerebral cortex cells300511,3929Illumina HiSeq[36]PetropolousHuman preimplantation embryo cells152921,7495Illumina HiSeq[37]YANHuman early embryos and embryonic stem cells9020,2147Illumina HiSeq[38]mBladderMouse bladder tissue cells274620,67016Microwell-seq[39]KleinMouse embryonic stem cells271724,0474InDrop[40]MouseMouse bladder tissue cells210020,67016Microwell-seq[39]PBMCHuman peripheral blood mononuclear cells432016,635810X–ChenAdult mouse hypothalamus cells12,08917,55046Drop-seq[41]Dataset PBMC is obtained from https://support.10xgenomics.com/single-cell-gene-expression/datasetsDataset mPCTC is obtained from GSE51372 Details of the 18 evaluated datasets Dataset PBMC is obtained from https://support.10xgenomics.com/single-cell-gene-expression/datasetsDataset mPCTC is obtained from GSE51372 Competing methods and evaluation metrics We conduct a comparative analysis of our proposed method against several representative cell clustering methodologies, including ScCAEs [22], SLMNF [21], scNAME [42], scDSC [23], scDCCA [15], scDFC [16], scGAC [24], scMAE [43], scziDesk [44], scDeepCluster [18], scAce [20], and scGMAAE [45]. To ensure a fair comparison, we either directly cite the best results reported in the related works or retrain the models using the released code and follow the optimal parameter settings as described in their respective publications. The summary of the competing methods can be found in Table 2. Table 2Summary of the compared baseline methodsAlgorithmLanguageDescriptionLinkReferencesScCAEsPython/RDeep learninghttps://github.com/gushenweiz/scCAEs[22]SLMNFMatlabClusteringhttps://github.com/xkmaxidian/SLNMF[21]scNAMEPythonContrastive clusteringhttps://github.com/aster-ww/scNAME[42]scDSCPythonDeep learninghttps://github.com/DHUDBlab/scDSC[23]scDCCAPythonContrastive clusteringhttps://github.com/WJ319/scDCCA[15]scDFCPythonFusion clusteringhttps://github.com/DayuHuu/scDFC[16]scGACPythonClusteringhttps://figshare.com/articles/software/scGAC/19091348?file=33926987[24]scMAEPythonClusteringhttps://zenodo.org/records/10465991[43]scziDeskPythonDeep learninghttps://github.com/xuebaliang/scziDesk[44]scDeepClusterPythonDeep learninghttps://github.com/ttgump/scDeepCluster[18]scAcePythonClusteringhttps://github.com/sldyns/scAce[20]scGMAAEPythonClusteringhttps://github.com/WHY-17/scGMAAE[45] Summary of the compared baseline methods To evaluate the clustering performance, we adopt three widely used metrics: Accuracy (ACC), Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI). Higher values of ACC, NMI, and ARI indicate better clustering performance. Cell type identification results The cell type identification performance of various scRNA- seq clustering algorithms across 18 scRNA-seq datasets is presented in Table 3. Our method consistently outperforms other competing methods in nearly all datasets, delivering stable results across all evaluation metrics. Specifically, our method achieves the highest scores on 10 datasets, including Human ESC, mPCTC, Human brain, Zeisel, Li cell, Leukemia, Petropolous, Klein, PBMC, and Chen. These results demon- strate its robustness and accuracy in cell clustering across diverse platforms, tissues, and organisms. Notably, our method performs exceptionally well on the Chen dataset, which con- tains over 10,000 cell samples, highlighting its capability in handling large-scale data. Table 3ACC, NMI and ARI values of 12 competitive methods in 18 scRNA-seq datasetsDatasetMetricsScCAESLMNFscNAMEscDSCscDCCAscDFCscGACscMAEscziDeskscDeepClusterscAcescGMAAEOursHuman_ESCACC83.9953.4475.5483.0182.6182.0282.5182.7176.1280.35 91.4572.29100NMI92.5966.0889.6189.8388.4889.8891.6591.7589.8288.92 96.6881.8100ARI78.2639.5372.6775.576.3875.1782.5177.1572.8175.03 93.9765.85100mESCACC33.339.0237.4147.3965.8547.5541.647.5549.3 72.3850.895074.91NMI–2.6747.9315.3123.6517.5710.6612.6713.4730.0226.120.35 33.84ARI–0.50.210.0625.1914.545.559.89 38.0134.2431.3214.6539.02mPCTCACC70.5449.1167.8531.2560.71 72.3266.9662.564.2850.8958.0358.0377.68NMI 71.7936.2961.618.8662.2964.4670.5355.5958.5950.4451.0271.1375.23ARI62.2516.1750.511.1244.662.0258.6442.949.7335.33 63.1252.4565.62Human_brainACC83.8169.2969.2842.8664.5278.57 88.0982.3869.7662.8672.3879.5288.33NMI 81.8859.5571.730.4563.6179.681.7174.9763.4172.6268.1372.0385.68ARI 82.0149.5358.910.7758.8577.9180.3466.5652.2659.1670.3567.5686.79Human_pancreasACC92.7552.7286.2183.8860.4178.2155.4696.3963.4650.1471.4285.09 95.98NMI95.0744.3184.8673.4262.0379.8274.0192.8562.2468.7571.1183.01 92.97ARI 95.8152.7281.1177.3249.5977.5152.6196.1778.4546.7477.0681.296.37Li_cellACC10065.2488.470.0580.9374.3383.0610089.4269.8867.5583.6100NMI10065.6688.472.1375.2682.0790.7810092.3978.6853.3289.33100ARI10040.0291.560.2873.5865.9681.310089.6165.274.586.69100Li_NMACC45.1145.8662.445.4947.1446.2448.49 54.8842.3527.0725.1843.9873.68NMI 44.9815.0446.911.9622.4521.9327.8453.239.3413.684.5841.3255.85ARI20.441.04 33.40.317.0215.1419.931.5416.651.6925.422.0656.3LekemiaACC23.0257.0336.5731.9747.8323.27–49.136.2342.7135.5131.4571.87NMI2.5241.0119.3713.7628.060.43–31.1120.1239.6233.0318.2259.45ARI4.8336.6510.75.324.890.5–22.1112.7427.0347.9213.3461.66SpleenACC67.0188.3480.6895.0455.9489.9164.5575.6163.4465.9946.29 95.3895.52NMI52.3452.4460.48 81.2945.7174.0448.9669.9944.7856.7921.8279.3882.06ARI74.8265.3266.7992.8533.8986.4533.4454.0461.0341.1348.5989.3 90.85Time CourseACC72.5650.23 81.2473.0579.7955.6170.9376.0863.9852.3178.670.9391.55NMI71.9163.47 85.1472.5171.6662.376.2279.6462.9560.4464.9475.2689.08ARI56.8236.4471.1161.3262.4837.363.8269.1251.1835.5482.2564.57 80.62ZeiselACC 87.7251.2579.5340.863.3661.3688.6182.6678.0865.1978.370.7191.38NMI72.8541.2568.3815.0855.8452.0175.3171.0863.4871.26 76.0466.0780.61ARI76.1833.0266.66.1944.9336.99 81.5970.2461.7358.7175.6560.786.1PetropoulousACC51.4750.88 74.4930.4154.2859.2570.1767.2959.0150.6947.9457.0976.85NMI46.6235.9861.683.5544.34 63.3359.7649.4844.4941.8323.0347.8764.54ARI23.9125.55 49.32.1428.8446.3544.835.9228.330.7538.6933.6452.54YANACC63.33–72.2261.1144.4480 77.775.5564.8171.1164.4464.4478.89NMI74.55–77.8255.4744.383.186.1680.6878.1176.4158.877.44 83.22ARI54–64.8734.0525.379.5683.3868.5159.7561.1566.1656.06 80.72mBladderACC47.9244.6864.9657.9460.9656.4447.7 77.0971.0863.5872.8365.6581.17NMI53.3749.2270.2456.5867.6361.9562.22 77.9870.6373.6662.7967.5878.68ARI47.9229.4849.4146.3451.8443.4232.9865.2956.1148.5777.3654.56 70.06KleinACC93.5687.2384.3180.985.0959.7782.0793.1586 98.3492.7471.4798.38NMI81.118479.6758.8281.0565.682.1481.3177.83 93.3690.9268.0893.38ARI84.9982.3379.6765.4180.949.8180.718479.5996.2590.5567.8796.25MouseACC75.4346.163.8154.2472.9549.1953.76 76.0964.6661.6273.973.3377.48NMI77.3849.3971.8351.9971.565.5965.2578.2368.9873.7664.9372.4178.23ARI63.6629.8151.1337.3556.940.338.43 67.1648.8747.8676.2459.2261.35PBMCACC 88.1653.6677.4884.2280.6755.928087.0977.6463.5982.8377.1891.36NMI 82.2537.8776.0772.2475.1458.1175.8581.7576.2969.4871.0271.4782.56ARI82.8453.6666.871.0378.2541.8767.59 83.170.3552.4170.2970.5383.63ChenACC 75.3242.9651.4634.0450.99––43.7827.1434.3961.2373.9278.4NMI 77.7434.0969.5463.253.95––69.0361.764.5967.1276.1677.78ARI80.9717.8951.8339.6844.01––36.0121.4626.6945.59 81.9385.29Highest value for each dataset is in bold. The second highest value is underlined. ‘–’ denotes the value is unavaliable) ACC, NMI and ARI values of 12 competitive methods in 18 scRNA-seq datasets Highest value for each dataset is in bold. The second highest value is underlined. ‘–’ denotes the value is unavaliable) For the remaining datasets, although our method does not always surpass all baseline methods across every metric, it consistently ranks among the top three, showcasing strong stability across datasets where performance is slightly below the leading method. While no single clustering method excels in every scenario due to the diversity of biological datasets, our method demonstrates superior performance in nearly all cases, achieving the best average scores in terms of ACC, NMI, and ARI. This suggests that our approach is not only robust but also highly accurate in cell clustering across different platforms, tissues, and organisms. Additionally, we present UMAP visualizations of the learned clustering representations to provide an intuitive biological interpretation of the clustering results, as shown in Fig. 2. The learned representations from our method are obtained by applying Singular Value Decomposition (SVD) on the learned similarity graph. To assess the performance of our method, we compare it with several competitive approaches, including scMAE, scAce, scGAMME, scDCCA, and scDSC. The results demonstrate that our method effectively distinguishes between different cell types. The clear clustering structure observed is advantageous for clustering tasks, as it facilitates the separation of distinct clusters and enhances the accuracy of the clustering outcomes. In contrast, scMAE and scDCCA struggle to identify well-defined clusters, particularly in the spleen and Petropolous datasets. These visualization results demonstrate that our method can accurately cluster various cell types, concentrating similar cells within clusters while effectively separating different cell types. UMAP visualization of the cell embeddings learned by ours method and comparative methods. The colors represent the clustering labels assigned by each method Model analysis Visulization of affinity matrix The quality of the learned affinity matrix is pivotal to the clustering performance, as it directly influences the delineation of distinct cell groups. To assess this, we present the affinity matrix in Fig. 3, where rows and columns are arranged according to the subtype labels obtained from clustering. The affinity matrix learned by our method for Human ESC , Li cell, mBladder, Time Course and Mouse datasets The visualization reveals that our method produces an affinity matrix with a pronounced block-diagonal structure and reduced noise. This clearly indicates strong intra-cluster similarity and weak inter-cluster connections, confirming that our approach yields robust clustering results and reliable cell type identification in scRNA-seq data. The effectiveness of Wasserstein-regularized self-representation Tn our method, the OT coupling is derived from the self-representation matrix , while in turn guides and refines , forming a mutually reinforcing optimization process. To assess the effectiveness of the OT strategy in self-representation learning, we set for simplicity and examined the effect of varying on both the self-representation matrix and overall clustering accuracy. Figure 4 summarizes the resulting clustering performance. The results show that our method achieves stable clustering performance across a wide range of , demonstrating robustness to the choice of this parameter. Clustering accuracy of the Wasserstein regularization weight on three datasets Robust analysis To further evaluate the robustness of our model under extremely sparse conditions, we investigate the effect of high dropout rates during the data augmentation process on clustering performance. Specifically, we randomly mask portions of gene expression values with different masking ratios to simulate varying degrees of incompleteness of the data. We then assess the performance of the clustering in each setting. As illustrated in Fig. 5, the clustering accuracy gradually decreases as the dropout rate increases, which is expected since more information is lost. However, our method maintains satisfactory performance even at extremely high dropout rates (up to 70%), demonstrating its strong resilience to missing data and its ability to preserve discriminative structure despite severe input degradation. ACC,NMI,ARI of the proposed method under progressively higher gene-masking ratios (dropout probabilities) on Human_ESC, Human_brain, and Time_course datasets Even when the masking ratio reaches 0.7 (mimicking extremely sparse expression profiles), our method still achieves satisfactory performance, demonstrating the robustness of the proposed framework against severe dropout variability. Convergence analysis The convergence behavior of our method was evaluated by plotting learning curves that depict the loss function values over training iterations for several datasets, including Mouse, Human_pancreas, Leukemia, Human_ESC, and mESC. In these plots, the x-axis represents the number of epochs, while the y-axis shows the scaled loss values, normalized between 0 and 1, with each dataset distinguished by a unique color. As illustrated in Fig. 6, the loss values decrease steadily across all datasets, indicating that our model converges effectively and efficiently. This consistent decline in loss demonstrates the robustness of our initialization strategy and the model is able to rapidly achieve satisfactory clustering performance. The normalized loss across epochs for Mouse, Human pancreas, Leukemia, Human ESC, and mESC Biological analysis Cell type assignment To visually assess the effectiveness of cell type assignments, As shown in Fig. 7, we employed Sankey diagrams to illustrate the correspondence between clustering results from various methods and the true cell types. These diagrams can provide a clear visualization of how cells are classified by each method in relation to their actual identities. Sankey plots of clustering results and true cell types for ours method and comparative methods. For each subplot, the left side represents the clustering labels generated by each method, while the right side represents the true cell types Our proposed method exhibits superior accuracy in assigning cells to their correct types compared to existing approaches. Notably, several methods struggle to distinguish major cell types in complex datasets. For example, in the Zeisel dataset, certain techniques incorrectly group interneurons, S1 pyramidal cells, and CA1 pyramidal cells into a single cluster, failing to resolve these distinct populations. In contrast, our method successfully differentiates these cell types, demonstrating its robustness in managing intricate scRNA-seq datasets. Furthermore, our approach excels in detecting rare cell types with limited sample size, a critical aspect of comprehensive scRNA-seq data analysis. For instance, our method can accurately identify natural killer (NK) cells, smooth muscle cells characterized by elevated Acta2 expression, and dendritic cells marked by high Cd74 expression, among others in mBladder data. This ability to resolve rare populations enhances the biological insights derived from scRNA-seq data.Overall, the Sankey diagrams verify the accuracy and reliability of our method in cell type assignment, highlighting its advantages over existing clustering techniques. Differential expression analysis Differential gene expression (DGEs) is essential to understand the biological differences between two or more sample groups. To verify the superiority of our proposed method, we selected the ‘Human ESC’ dataset and further examined the ’Time Course‘ dataset to determine whether our method benefits downstream analysis. For ‘Human ESC’, there are seven different cell subpopulations, including neuronal progenitor cells (NPCs), definitive endoderm derivative cells (DEDs), endothelial cells (ECs), trophoblast-like cells (TBs), undifferentiated H1 and H9 ESCs, and foreskin fibroblasts (HFFs). In the ‘Time Course’ dataset, the cells are collected at six different time points (0, 12, 24, 36, 72, and 96 h) during the developmental trajectory of H9 ESCs differentiating towards DEDs. For each method, we select the top 50 DEGs of each cluster and compare their overlap to find out whether we can annotate every cluster correctly to a known cell type in the golden standard. (Top 50 DEGs in different cell types in golden standard using the Wilcoxon Rank Sum test can be seen in Fig. 9). The similarity calculated in Fig. 8 equals the number of overlapped DEGs divided by 50. Each column represents a cluster acquired by a specific method, while each row represents a known cell type regarded as the golden standard. The higher heatmap value shows that it holds the highest similarity in DGEs. Marker gene analysis in the “Human ESC” dataset. Ranking of genes in different cell types using the Wilcoxon Rank Sum test Overlap of top 50 DEGs in clusters detected by our methods, scMAE and scNAME with gold standard cell types. A Marker gene analysis in the ‘Human ESC’ dataset. B Marker gene analysis in the ‘Time Course’ dataset As shown in Fig. 8, our method accurately maps each cell type to a distinct cluster in both datasets. Specifically, it achieves similarity scores of 0.9 for most clusters, successfully distinguishing even closely related H1 and H9 ESCs, which are assigned to clusters 0 and 4, respectively. Additionally, in the ‘Time Course’ dataset, our method clearly delineates H9 ESCs across their developmental stages, with each time point corresponding to a unique cluster. These results demonstrate that our approach reliably annotates clusters to specific cell types based on highly similar gene expression profiles. In contrast, the second-best method, scMAE, exhibits notable limitations. For the ‘Human ESC’ dataset, scMAE assigns both H1 and H9 ESCs to cluster 3 and fails to assign any cell type to cluster 6. In the ‘Time Course’ dataset, scMAE merges time points 12 and 24 into cluster 1 and leaves another cluster without a corresponding cell type. Similarly, scNAME struggles to differentiate H1 and H9 ESCs, grouping both into cluster 0, and in the developmental case, it conflates multiple time points into fewer clusters. These shortcomings highlight that only our method consistently and accurately annotates each cluster to a known cell type across both datasets. Conclusion In this study, we introduce a novel deep subspace clustering approach for cell type identification in scRNA-seq data. Our method addresses the challenges posed by noise and sparsity inherent in scRNA-seq datasets by employing a robust self-representation learning framework. This framework effectively captures and learns the underlying cluster structure, enabling precise cell type delineation. To optimize this process, we integrate a structure-guided strategy with an optimal transport algorithm, which collectively enhance the robustness of the subspace clustering framework. Consequently, our approach achieves more accurate and reliable cell clustering outcomes. Experimental evaluations across a diverse set of real scRNA-seq datasets demonstrate that our method outperforms several state-of-the-art clustering techniques in both clustering accuracy and interpretability. Additionally, comprehensive model and biological analyses validate the efficacy of our approach in accurately annotating cell types and revealing biologically meaningful insights. Acknowledgements This work was supported in part by the Scientific Research Innovation Capability Support Project for Young Faculty (ZYGXQNJSKYCXNLZCXM-H8), in part by the GuangDong Basic and Applied Basic Research Foundation (Project No.2023A1515030154, 2025A1515011692, 2024A1515011437), in part by the Fundamental Research Funds for the Central Universities (Project No. 2024ZYGXZR077), in part by TCL Science and Technology Innovation Fund (Project No. 20231752), in part by the Research Grants Council of the Hong Kong Special Administration Region (Project No. CityU 11206622). Author contributions X.Z. and X.W., as co-first authors, played major roles in designing the study and drafting the manuscript. C.L. and W.S., supervised the project and have prepared the manuscript. P.X., S.W., and H.W. contributed to reviewed the manuscript. Data availability Code replication on Github: https://github.com/a1194869892/Robust.git The dataset can be downloaded here: https://stumail-my.sharepoint.cn/:f:/g/personal/22xyzhou2_stu_edu_cn/EnnnxeAI43JAtD-MVJmecjkBhygRLzbVDLZSa24sKJHGHA The entire dataset is available for download from GEO under accession number GSE51372 at: https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi Declarations Ethics approval and consent to participate Not applicable, as this study did not involve human or animal subjects. Consent for publication Not applicable, as no identifiable data are included in this publication. Competing interests The authors declare no conflict of interest. Footnotes Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Xianyong Zhou, Xindian Wei have contributed equally to this work. References Associated Data Data Availability Statement Code replication on Github: https://github.com/a1194869892/Robust.git The dataset can be downloaded here: https://stumail-my.sharepoint.cn/:f:/g/personal/22xyzhou2_stu_edu_cn/EnnnxeAI43JAtD-MVJmecjkBhygRLzbVDLZSa24sKJHGHA The entire dataset is available for download from GEO under accession number GSE51372 at: https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi