Ensemble clustering: A practical tutorial Authors: Caroline X. Gao, Shengqi Wang, Ye Zhu, Myriam Ziou, Shu Mei Teo, Catherine L. Smith, Derek Chiu, Aline Talhouk, Mengmeng Wang, Wenhua Yu, Sue M. Cotton, Dominic Dwyer Published: 2026 ABSTRACT Cluster analysis is an explorative analytical method, serving as a critical tool in psychology, psychiatry, and related fields to map heterogeneous data into meaningful subgroups. Despite their extensive historical use, traditional clustering techniques suffer from a lack of stability, robustness, and generalisability. These issues stem from the inherent difficulties of the clustering optimisation problem as well as the stochastic nature of algorithm optimisers. To address these challenges, we demonstrate the use of methods utilising ensemble learning techniques to combine clustering results from different algorithms, model specifications, and/or sampled sub-datasets to form a single, more reliable consensus of clustering solutions. We detail ensemble clustering principles and variations in base clustering generation models and ensemble methods. More importantly, detailed introductions in existing R libraries and practical examples using R code are provided to guide users in both implementing and optimising ensemble clustering models. As a practical tutorial, we then include simulation studies of real-world data to demonstrate the substantial benefit of ensemble clustering compared with single-run clustering models. The resources presented here will enable researchers to apply advanced clustering techniques to decompose heterogeneous and complex psychological data into stable subgroups.Supplementary InformationThe online version contains supplementary material available at https://doi.org/10.3758/s13428-026-03158-y. Introduction Categorising groups of similar objects, elements, or individuals is a cornerstone of scientific advancement, but the process can be influenced by human bias (Yoon, 2009). Statistical clustering arising in the mid-twentieth century promised to reduce this bias by automatically grouping data points based on their similarity (see terminology definition in Table S1). The term “cluster analysis” was first introduced in psychology, dating back to 1939, when R.C. Tryon (1939) introduced it to identify “clusters” of variables. This approach remains a foundational method for exploring symptom clusters and contributing to disease nosology (Forbes et al., 2021, 2024). Over time, the application of clustering models in health shifted towards exploring the heterogeneity of individuals rather than just variables, enabling researchers to identify distinct subgroups of people based on shared behavioural, biological, or clinical characteristics (Cattell, 1943). To date, clustering models are widely used in psychology, psychiatry, and other related fields for a range of purposes, such as understanding the heterogeneity of disorders using data from clinical profiles, genetic risks, brain pathophysiology, and blood markers (Beijers et al., 2019; Dwyer et al., 2022; Marquand et al., 2016); evaluating treatment response heterogeneity (Bondar et al., 2020; Case et al., 2011); identifying clinical complexity profiles for service optimisation (Cotton et al., 2022; Gao et al., 2023a, 2023b); exploring symptom profiles (Bondar et al., 2020; Oladottir et al., 2022); extracting biological/pathological markers (Franklyn et al., 2022; Sæther et al., 2023); and mapping longitudinal patterns (Hall et al., 2019; Pierce et al., 2021). Over 60,000 articles have been published to date and more than 5,000 in 2024 alone.1 Despite this remarkable volume of research, their impact on research, practice, and policy remains severely limited (Feczko & Fair, 2020; Milligan & Cooper, 1987; Thorndike, 1953). Several core issues hinder the greater impact of clustering studies, such as a lack of clustering stability (the ability of results to remain consistent under repeated runs), robustness (resilience to noise in the data), and generalisability (applicability to different datasets and conditions); see detailed discussion elsewhere (Gao et al., 2023a, 2023b). Commonly used methods in psychology and psychiatry to address these issues have been ineffective (e.g., multiple start-points for k-means), evident from their statistical sensitivity to distributions of data points, minor data perturbations, and choice of clustering algorithms (Kleinberg, 2002; Milligan, 1980; Shireman et al., 2016; Steinley, 2003; Wolpert & Macready, 1997). Such failure is theorised to be related to an impossibility of simultaneously optimising across multiple criteria (e.g., the “impossibility theorem”, no clustering method can satisfy all desirable properties at once) (Kleinberg, 2002) in the context of a problem that does not have an exact solution (Steinley, 2003). Most algorithms rely on assumptions and trade-offs between different desirable properties. One major downstream implication is the replication crisis. Identified subtypes often vary substantially across studies; and when broadly similar patterns emerge, they frequently reflect arbitrary cut-offs along one severity dimension rather than providing meaningful insight into true heterogeneity (Brucar et al., 2023; Yu et al., 2026). For example, despite decades of research, current evidence does not support reproducible, biologically valid, and clinically meaningful neuroimaging subtypes for mental illnesses that are broadly agreed upon (Beijers et al., 2019; Brucar et al., 2023; Yu et al., 2026). These issues that continue to hamper research in psychology and psychiatry have been more effectively addressed in other medical fields (e.g., oncology and tumour subtyping), resulting in expanded impact and real-world translation (Fred & Jain, 2002, 2005; Monti et al., 2003; Polikar, 2006; Strehl & Ghosh, 2002; Topchy et al., 2004). Ensemble clustering methods2 are the primary driver, which combine clustering solutions from different data distributions, clustering algorithms, and optimisations before reaching a consensus on the final solution (Fred & Jain, 2002, 2005; Monti et al., 2003; Polikar, 2006; Strehl & Ghosh, 2002; Topchy et al., 2004). As a meta-method, ensemble clustering models not only offer more stable and robust solutions but also help in determining the optimal number of clusters from a stability perspective (Monti et al., 2003; Șenbabaoğlu et al., 2014). Another advantage of ensemble clustering is that it can facilitate the detection of complex decision boundaries that may not have been present in the underlying single-run clustering solutions (Brière et al., 2021; Fu et al., 2020; Jain, 2010; Rappoport & Shamir, 2018; Xu et al., 2013; Yang & Wang, 2018). Despite the availability of generalist methodological reviews (Boongoen & Iam-On, 2018; Golalipour et al., 2021; Nie et al., 2023; Vega-Pons & Ruiz-Shulcloper, 2011) and tutorials designed for machine learners and statisticians (Abbasi et al., 2019; Chiu & Talhouk, 2018), there is a critical lack of conceptual and practical guides to ensemble clustering specifically designed for an audience who are not familiar with advanced machine learning and statistical models, particularly in psychology and psychiatry, where clustering methods are frequently applied. This tutorial aims to fill this gap by providing a review of theoretical principles as well as practical implementations in ensemble clustering methods. Focusing on the R programming language, which is commonly used in psychology and psychiatry research, this practical tutorial presents (1) a brief overview of relevant clustering methods, (2) an introduction to ensemble clustering principles with a summary of R packages, (3) examples of applications with detailed steps and code in the R language, and (4) a simulation study comparing ensemble clustering with other models to emphasise the urgent need for the use of these methods in psychology and psychiatry. This tutorial also requires familiarity with basic clustering methods; therefore, we recommend readers consult our earlier methodological overview (Gao et al., 2023a, 2023b), which provides a detailed description of different clustering models as well as applications in mental health research. Types of clustering methods Since the 1960 s, numerous clustering algorithms have been developed. Here we provide a brief description of a few main models (see Fig. 1) to aid in understanding their use in ensemble clustering. The most common clustering method used in psychology and psychiatry is model-based clustering, which assumes that the data originate from latent subpopulations, each having different distributions (e.g., finite mixture models such as latent class analysis [LCA] and latent profile analysis [LPA]) (Clatworthy et al., 2005). Another popular family of methods is centre-based partitioning clustering (e.g., k-means), which iterates partitions subgroups based on distances between data points and cluster centres. A deterministic approach is hierarchical clustering, which constructs a tree of clusters (as a dendrogram) by either iteratively merging similar data points or dividing dissimilar subclusters.Fig. 1Illustration of main types of clustering algorithms Illustration of main types of clustering algorithms Several more modern clustering methods have also been widely used. For example, density-based clustering is designed to address problems associated with outliers by forming clusters in high-density regions and excluding outliers in low-density regions. Graph-based clustering refers to a range of diverse methods designed to segment networks of interconnected elements (e.g., social networks) and includes the detection of groups of similar points as modules or spectra (e.g., spectral clustering), or by considering pathways between network elements. Principles and processes of ensemble clustering Ensemble clustering methods refer to meta-learning methods for pooling, aggregating, and integrating results from different clustering algorithms or different runs of the same algorithm (with different data, parameter settings, or initialisations) to form a consensus (i.e., a stable grouping pattern that recurs across clustering solutions, rather than complete agreement between them). Conceptually, they are analogous to ensemble methods in supervised machine learning, in which predictions from multiple models (or repeated runs of the same model) are combined to reduce sensitivity to the limitations of any single model, improve robustness and avoid overfitting (Polikar, 2006). In ensemble clustering, however, the goal is to improve the stability and reproducibility of subgroup identification rather than prediction accuracy. Although many clustering ensemble methods have been proposed, they all follow a similar framework (see Fig. 2). Ensemble clustering generally involves two stages: generating base clustering (BCs, sometimes also referred to as base clusters) and obtaining consensus. The ensemble processes are commonly designed to achieve two goals: to obtain robust clustering solutions or to combine clustering results from different sources. The latter is known as multi-view ensemble clustering,3 where for example separate clustering solutions can be obtained from brain imaging, qualitative interviews, video or audio recordings, genetic data, and blood biomarkers, which can be integrated into a single solution (Gao et al., 2023a, 2023b). As the processes of multi-view ensemble clustering are slightly different, they are not the focus of this tutorial. Interested readers can find further information on multi-view clustering elsewhere (Chao et al., 2021; Fu et al., 2020; Yang & Wang, 2018; Zhang, 2022).Fig. 2Common framework of ensemble clustering Common framework of ensemble clustering Generating BCs The relatively simple first step of ensemble clustering involves generating many different clustering solutions, known as BCs. BCs can be obtained by running different models (e.g., different random seeds, parameters, or algorithms) on the same data or the same model on different data (e.g., random samples of observations and/or variables, variables randomly projected to different subspaces) or combinations of different approaches (see Fig. 3). In practical tasks, random sampling of observations and/or variables is often used due to the ease of operationalisation and interpretation. Dimensionality reduction is often required prior to generating BCs, particularly with large numbers of variables or strong intercorrelations to improve the performance of BCs (Gao et al., 2023a, 2023b).Fig. 3Simple illustration of main methods for generating diverse base clusters. Note: Blue colour indicates selected observations or variables; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${C}_{1}, {C}_{2}, \mathrm{a}\mathrm{n}\mathrm{d}\, {C}_{3}$$\end{document}C1,C2,andC3 represent three base clusterings (BCs), each with two partitions labelled as 1 and 2 Simple illustration of main methods for generating diverse base clusters. Note: Blue colour indicates selected observations or variables; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${C}_{1}, {C}_{2}, \mathrm{a}\mathrm{n}\mathrm{d}\, {C}_{3}$$\end{document}C1,C2,andC3 represent three base clusterings (BCs), each with two partitions labelled as 1 and 2 In general, BCs are recommended to be both diverse (presented with differences in clustering solutions) and of high quality (optimal at separating observations that can be measured using different criteria) (Fern & Lin, 2008; Kuncheva & Hadjitodorov, 2004; Li et al., 2019). Achieving these two objectives simultaneously is challenging because the models that increase diversity (e.g., aggressive sub‑sampling, randomising algorithmic) can also degrade the quality of individual partitions (e.g., some clustering models do not fit data well), and conversely, optimisation of quality often drives all BCs towards a similar solution, reducing diversity. However, their relative importance in enhancing ensemble clustering effectiveness is not universally agreed upon, making it a key domain of ongoing research in the field (Golalipour et al., 2021; Li et al., 2019). Achieving consensus The second stage, obtaining consensus, however, is more complex than generating BCs, and various methods have been proposed. The simplest approach is to use a majority voting method among the partitions in BCs (Zhou & Tang, 2006). For instance, if BCs categorise observations into three clusters (e.g., low, medium, and high), the consensus classification for a specific observation would be the category to which the majority of BCs assign that observation. Clustering models normally return random cluster labels (e.g., the low, medium, and high groups in one model are labelled as first, second, and third, which can change to third, second, and first in another model). Therefore, BCs will need to be aligned or relabelled (similar clusters given the same label) before they can be combined (Zhou & Tang, 2006). However, when the BCs are very diverse or generated from a method such as random-k, majority voting cannot be conducted. To avoid the relabelling step, address the practical challenges of majority voting, and improve consensus quality, more complex methods are needed. A commonly applied technique is to first summarise BCs and then carry out a second-stage clustering model (see Fig. 4).Fig. 4Simple illustration of several ensemble clustering methods. Note: Top panels illustrate three base clustering (BC) solutions derived from different algorithms or modelling conditions (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${C}_{1}, {C}_{2}, \mathrm{a}\mathrm{n}\mathrm{d}\, {C}_{3}$$\end{document}C1,C2,andC3) with two partitions (labelled as 1 and 2) for five data points. Middle panels represent possible cluster summarising matrices that can be used to integrate BC solutions as well as their associations (e.g., how binary cluster association can be derived from cluster labels). Bottom panels demonstrate possible modelling approaches to summarise cluster results and obtain the final ensemble clustering solution Simple illustration of several ensemble clustering methods. Note: Top panels illustrate three base clustering (BC) solutions derived from different algorithms or modelling conditions (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${C}_{1}, {C}_{2}, \mathrm{a}\mathrm{n}\mathrm{d}\, {C}_{3}$$\end{document}C1,C2,andC3) with two partitions (labelled as 1 and 2) for five data points. Middle panels represent possible cluster summarising matrices that can be used to integrate BC solutions as well as their associations (e.g., how binary cluster association can be derived from cluster labels). Bottom panels demonstrate possible modelling approaches to summarise cluster results and obtain the final ensemble clustering solution The cluster labels from BCs can be treated as categorical variables. Therefore, consensus can be obtained using a finite mixture model (Topchy et al., 2004; Yu et al., 2014) or other methods of clustering categorical variables such as k-mode (Chiu & Talhouk, 2018; Huang, 1997). The cluster labels can also be converted into a binary cluster association matrix using binary encoding (each cluster label is converted to multiple binary variables representing individual partitions), which can be used as input data for methods such as k-means, e.g., k-means-based consensus clustering (KCC) (Lin et al., 2023; Wu et al., 2015). The binary cluster association matrix can be transformed into a bipartite graph (a graph connecting clusters with observations; see Fig. S1 in Supplementary Material I) and consensus can be achieved with graph-based models (Yu et al., 2007). A hypergraph-partitioning algorithm (HGPA), for example, proposed by Strehl and Ghosh (2002) utilised this approach. The cluster labels can also be summarised by pairwise associations between observations (e.g., a co-association matrix, which measures how frequently pairs of data points are grouped together across BCs). The co-association matrix can be used directly as distance measures for a second-stage clustering model such as hierarchical clustering, which was one of the earliest and most widely employed methods, also known as evidence accumulation clustering (EAC) (Fred & Jain, 2002, 2005) or consensus clustering (Monti et al., 2003). The co-association matrix can also be converted to weighted graphs (nodes are the observations and the weights between two nodes are the corresponding values in the co-association matrix) or hypergraphs (observations represented by nodes connected by hyperedges representing the membership of the same cluster; see Fig. S1 in Supplementary Material I). Then the final consensus can be treated as a graph partitioning problem (Liu et al., 2015); for example, the cluster-based similarity partitioning algorithm (CSPA) applies a graph-based clustering model, METIS, on the co-association matrix (Strehl & Ghosh, 2002). As the co-association matrix only evaluates the similarity between pairs of observations but not between BCs, other similarity matrices have been proposed, e.g., connected-triple-based similarity (CTS) proposed in the link-based clustering ensemble (LCE) (Iam-On & Garrett, 2010; Iam-On et al., 2011). Several alternative consensus methods have been formulated. For instance, the meta-clustering algorithm (MCLA) involves consideration of each BC as a hypergraph. Its consensus function merges similar hyperedges (Strehl & Ghosh, 2002). Another approach for obtaining consensus from BCs is to estimate a mean (median) partition that minimises the disagreement of all BCs using methods such as aggregating partitions or refining cluster centroids (Bradley & Fayyad, 1998; Gionis et al., 2007; Jain, 2018). Adaptive clustering ensemble (ACE) is another method based on the binary clustering association matrix, which iteratively merges binary representations based on a similarity measure between clusters (Alqurashi & Wang, 2019). Recent developments also include ensemble clustering using deep learning (Cai et al., 2021; Ma et al., 2016). Detailed reviews and summaries of diverse ensemble clustering models are available in the machine-learning literature (Boongoen & Iam-On, 2018; Ghaemi et al., 2009; Golalipour et al., 2021; Vega-Pons & Ruiz-Shulcloper, 2011; Xanthopoulos, 2014). Choosing BC algorithm(s) and ensemble methods BCs can be generated through a variety of sampling and modelling techniques, tailored to the specific characteristics and dimensions of the dataset (e.g., types and distributions of variables, number of observations) under consideration. Resampling of observations is commonly applied in larger samples, and variable or feature resampling strategies are particularly useful in high-dimensional settings. When choosing algorithms for BCs, it is important to consider the dataset’s specific characteristics, including data size, variable types, and the anticipated configurations of clusters. In practice, k-means is the most commonly used clustering algorithm due to its low computational demands; however, it is not without its limitations (e.g., assumption of spherical clusters and sensitivity to outliers). In psychological and mental health research, model-based clustering (e.g., LCA) may also be a strong candidate BC algorithm, as it can accommodate subgroups that plausibly arise from underlying generative processes. Multiple BC algorithms can also be used and tested to improve the robustness of the modelling approach. The choice of ensemble methods can also depend on the features of the data. Majority voting is the earliest and most straightforward approach. However, it is most appropriate when BCs produce highly similar cluster assignment results. It also does not take full advantage of ensemble methods, i.e., integrating diverse results from base algorithms (weak learners) to achieve high performance (Polikar, 2006). The co-association matrix is the most commonly used method due to its interoperability and flexibility. When displayed using heatmaps, the co-association matrix can provide a direct visual presentation of cluster membership, which can be interpreted as relative stability when choosing the number of clusters (Monti et al., 2003). There are also flexible ways to decompose the co-association matrix for consensus (e.g., as a distance matrix or graph). However, it can be difficult to calculate with an increasing number of observations. This can be addressed by using the binary cluster association matrix with big data. Although the literature is sparse on benchmarking different ensemble clustering algorithms, limited evidence suggests that the accuracy and robustness of most of these techniques are high and that differences may be due to specific characteristics (e.g., dimensionality, distribution, and size) of the data (Ghaemi et al., 2009; Wu et al., 2018). We provide high-level considerations and recommendations for establishing the whole ensemble clustering workflow in Fig. 5, with possible cross-validation (CV, a method that repeatedly applies a model to resampled or partitioned data to improve its generalisability) procedures outlined in Fig. S3.Fig. 5Simple illustration of advisory ensemble clustering workflow with general modelling considerations and recommendations Simple illustration of advisory ensemble clustering workflow with general modelling considerations and recommendations Optimising parameters Substantial research effort has been focused on improving the performance of ensemble models, particularly by improving the quality and diversity of BCs (Abbasi et al., 2019; Berikov, 2014; Zhang, 2022). Diversity of BCs can be achieved through the sampling processes or different clustering models. For BC models requiring specifying the number of clusters (kk), diversity can also be achieved by specifying a larger kk or random-k. In these cases, kk is recommended to be equal to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sqrt{sample\, size}$$\end{document}samplesize or randomly selected between 2 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sqrt{sample\, size}$$\end{document}samplesize (Fred & Jain, 2005; Iam-On et al., 2011). However, the computational cost or complexity may increase for some models with increasing kk in large datasets (e.g., the dimensions and sparseness of the binary association matrix). Hence, random sampling and projection methods are often preferred for generating BCs. In such scenarios, it is advisable to use the same kk for BCs that is required for the final clustering results (Monti et al., 2003). Improving the quality of BCs can be achieved by selecting or weighting BCs with high quality as well as retaining diversity (Golalipour et al., 2021). The number of BCs (ensemble size) may also impact clustering performance depending on the features of the data. Generally, increasing ensemble size is associated with higher performance, particularly with diverse BCs (Fern & Brodley, 2003; Wu et al., 2018). Among the limited benchmarking studies, clustering quality normally plateaus for ensemble sizes between 50 and 100 (Fern & Brodley, 2003; Wu et al., 2015, 2018). In single-run clustering models, a range of internal validity indices, e.g., Silhouette index, Calinski–Harabasz (CH) index, Dunn index, and Gap index, have been developed to facilitate the choice of the optimal number of clusters (Gao et al., 2023a, 2023b), which do not generally agree with each other. The advantage of ensemble clustering is that BC stability indicators (e.g., heatmap of co-association matrix) can be used (Dwyer et al., 2020; Monti et al., 2003; Șenbabaoğlu et al., 2014). The optimal number of clusters would ideally demonstrate higher stability, greater overall consensus between pairs of BCs, and better overall fitting performance in the final clusters (see detailed explanation in the implementation section below with examples). In studies with a high explorative purpose, selecting the optimal number of clusters should also take into consideration the meaningfulness of clusters (Gao et al., 2023a, 2023b). Implementation of ensemble clustering models Traditional software packages, such as IBM® SPSS® Statistics and Stata, only allow for standard applications of clustering models, i.e., clustering models based on a single run of the algorithm. Although ensemble clustering models can be implemented via additional programming, they are substantially difficult due to the challenges in programming environments (e.g., operating multiple data objects simultaneously). As open-source statistical programming languages such as R and Python are becoming increasingly popular, the implementation of ensemble clustering models is now much more accessible. To date, there have been a range of R packages developed for ensemble clustering models, such as diceR, ConsensusClusterPlus, and clue (see Table 1, with the popularity of these packages approximated by frequency of downloads between 2015 and 2025 estimated in Fig. S2). Many of the existing packages were inspired by and followed the design principle of the ensemble clustering model proposed by Monti et al. (2003). Here we have also included brief information regarding other related packages that were developed specifically for a certain type of data or multi-view clustering. Table 1Ensemble clustering implementation in RMain R packagesReferencesdiceR can perform all the steps on the original dataset, from generating base clusters to consensus formation. It includes a large number of methods for base clustering as well as methods for visualisation of results that inform hypothesis testingMain functions: dice(), consensus_cluster(), consensus_matrix(), consensus_combine(), consensus_evaluate()Methods for sampling: Random sampling of observationsMethods for base clusters: Nonnegative matrix factorisation, hierarchical clustering, divisive analysis clustering, k-means, partition around medoids, affinity propagation, spectral clustering, Gaussian mixture model, co-clustering using multiple latent block model, self-organising map with hierarchical clustering, fuzzy c-means clustering, and hierarchical DBSCAN, as well as custom-defined clustering algorithmsEnsemble methods: Majority voting, k-modes (categorical variable clustering of cluster label), cluster-based similarity partitioning algorithm (CSPA, graph partitioning of co-association matrix), link-based cluster ensembles (LCE, spectral graph partitioning of co-association matrix), and LCA (categorical variable clustering of cluster label). In the diceR package, CSPA and LCE applied hierarchical clustering instead of the original graph models proposed by Strehl and Ghosh (2002) and Iam-On and Garrett (2010)(Chiu & Talhouk, 2018; Chiu et al., 2024)ConsensusClusterPlus is a simple package based on two functions only, one generating the base clusters and implementing consensus clustering, and the other calculating cluster-consensus and item-consensus. Several graphical features are incorporated in these two functionsMain functions: calcICL(), ConsensusClusterPlus()Methods for sampling: Random sampling of observations and/or variablesMethods for base clusters: Hierarchical clustering, partitioning around medoids, and k-means, as well as custom-defined clustering algorithmsEnsemble methods: Hierarchical consensus clustering based on the method developed by Monti et al. (2003), i.e., hierarchical clustering based on co-association matrix(Wilkerson & Hayes, 2010; Wilkerson & Waltman, 2020)clue forms consensus on a collection of partitions (soft or hard) or hierarchies (n-valued trees) by minimisation of dissimilarity between consensus candidates and base clusters (“optimisation approach”)Main functions: cl_consensus(), cl_dissimilarity(), cl_ensemble()Methods for sampling: Random sampling of observations and variation of parameters/algorithmsMethods for base clusters: Partitions or hierarchies must be created prior to using the package with the desired algorithm. Both soft/fuzzy and hard/crisp partitions are possibleEnsemble methods: Minimisation is done differently for partitions and hierarchies, with a range of possible methods. For partitions: fixed-point algorithm for soft/hard least squares Euclidian/median Manhattan consensus partitions, fixed-point algorithm for the “first model” from Gordon and Vichi (2001), greedy algorithm of Dimitriadou et al. (2002), sequential unconstrained minimisation techniques. For hierarchies: minimisation of Euclidean dissimilarity, sequential unconstrained minimisation technique with Manhattan dissimilarity, majority consensus tree of Margush and McMorris (1981). Custom-defined algorithms can be used for partitions and hierarchies(Hornik, 2005; Hornik & Böhm, 2023)clusterCons package performs all the steps on the original dataset, from generating base clusters to consensus formation. Results from different consensus clustering can be merged, considering a user-specified weighting. Visual comparisons can be performed between the number of clustersMain functions: cluscomp(), clrob(), memrob()Methods for sampling: Random sampling of observations and variation of parameters/algorithmsMethods for base clusters: Hierarchical clustering, divisive analysis, partitioning around medoids, and k-means, as well as custom-defined clustering algorithmsEnsemble methods: Hierarchical consensus clustering based on the method developed by Monti et al. (2003)(Simpson, 2022; Simpson et al., 2010)M3C package uses Monte Carlo simulation to simulate stability scores distribution, in order to inform the selection of optimal number of clusters (via hypothesis testing)Main functions: M3C()Methods for sampling: Random sampling of observationsMethods for base clusters: Partitioning around medoids, k-means, spectral clustering, hierarchical clusteringEnsemble methods: Monte Carlo reference-based consensus clustering based on Monti et al. (2003), regularised consensus clustering (addition of penalty term to the estimation of the optimal K). This model was developed to address the issue of existing methods (e.g., PAC) in identifying optimal clusters, which allows comparisons with null distributions using hypothesis testing(John & Watson, 2023; John et al., 2020)cola package internally using clue for assigning consensus labels. Incorporates top-values methods to select variables with highest variability as part of the pre-processing stage. It can also perform hypothesis testing to identify specificity of variables in one or several clusters (e.g., using permutation)Main functions: consensus_partition(), run_all_consensus_partition_methods()Methods for sampling: Random sampling of observations or variables and variation of algorithmsMethods for base clusters: Hierarchical clustering, k-means, spherical k-means, partitioning around medoids, and model-based clustering, as well as custom-defined clustering algorithmsEnsemble methods: Internal use of cl_consensus() from the clue package(Gu, 2023; Gu et al., 2021)Other related packages:coca: an extension of ConsensusClusterPlus, which focuses on multi-view clustering, using a multi-cluster-of-clusters analysis (COCA). The individual results from individual views were combined using an unweighted approach(Cabassi & Kirk, 2020a, 2020b)klic: multi-view clustering using the kernel learning integrative clustering (KLIC) method, which is similar to coca. klic uses the kernel k-means approach, which allows one to assign weights to different views, aimed at removing view-specific noise(Cabassi & Kirk, 2020b; Cabassi et al., 2020)SC3: specific consensus package for clustering single-cell RNA-Seq data. The consensus clustering approach is similar to the method proposed by Monti et al. (2003) except that BCs were generated from data with different methods of distance calculation and transformation(Kiselev, 2023; Kiselev et al., 2017)clusterExperiment: specific consensus package for clustering single-cell and other large gene expression data(Purdom & Risso, 2023; Risso et al., 2018)BCClong: Bayesian consensus clustering for multi-view longitudinal data using a joint modelling approach(Tan et al., 2022, 2023)Not all packages are in CRAN, and some packages will not be maintained regularly. Please check detailed package documentation when using these packages Table 1Ensemble clustering implementation in RMain R packagesReferencesdiceR can perform all the steps on the original dataset, from generating base clusters to consensus formation. It includes a large number of methods for base clustering as well as methods for visualisation of results that inform hypothesis testingMain functions: dice(), consensus_cluster(), consensus_matrix(), consensus_combine(), consensus_evaluate()Methods for sampling: Random sampling of observationsMethods for base clusters: Nonnegative matrix factorisation, hierarchical clustering, divisive analysis clustering, k-means, partition around medoids, affinity propagation, spectral clustering, Gaussian mixture model, co-clustering using multiple latent block model, self-organising map with hierarchical clustering, fuzzy c-means clustering, and hierarchical DBSCAN, as well as custom-defined clustering algorithmsEnsemble methods: Majority voting, k-modes (categorical variable clustering of cluster label), cluster-based similarity partitioning algorithm (CSPA, graph partitioning of co-association matrix), link-based cluster ensembles (LCE, spectral graph partitioning of co-association matrix), and LCA (categorical variable clustering of cluster label). In the diceR package, CSPA and LCE applied hierarchical clustering instead of the original graph models proposed by Strehl and Ghosh (2002) and Iam-On and Garrett (2010)(Chiu & Talhouk, 2018; Chiu et al., 2024)ConsensusClusterPlus is a simple package based on two functions only, one generating the base clusters and implementing consensus clustering, and the other calculating cluster-consensus and item-consensus. Several graphical features are incorporated in these two functionsMain functions: calcICL(), ConsensusClusterPlus()Methods for sampling: Random sampling of observations and/or variablesMethods for base clusters: Hierarchical clustering, partitioning around medoids, and k-means, as well as custom-defined clustering algorithmsEnsemble methods: Hierarchical consensus clustering based on the method developed by Monti et al. (2003), i.e., hierarchical clustering based on co-association matrix(Wilkerson & Hayes, 2010; Wilkerson & Waltman, 2020)clue forms consensus on a collection of partitions (soft or hard) or hierarchies (n-valued trees) by minimisation of dissimilarity between consensus candidates and base clusters (“optimisation approach”)Main functions: cl_consensus(), cl_dissimilarity(), cl_ensemble()Methods for sampling: Random sampling of observations and variation of parameters/algorithmsMethods for base clusters: Partitions or hierarchies must be created prior to using the package with the desired algorithm. Both soft/fuzzy and hard/crisp partitions are possibleEnsemble methods: Minimisation is done differently for partitions and hierarchies, with a range of possible methods. For partitions: fixed-point algorithm for soft/hard least squares Euclidian/median Manhattan consensus partitions, fixed-point algorithm for the “first model” from Gordon and Vichi (2001), greedy algorithm of Dimitriadou et al. (2002), sequential unconstrained minimisation techniques. For hierarchies: minimisation of Euclidean dissimilarity, sequential unconstrained minimisation technique with Manhattan dissimilarity, majority consensus tree of Margush and McMorris (1981). Custom-defined algorithms can be used for partitions and hierarchies(Hornik, 2005; Hornik & Böhm, 2023)clusterCons package performs all the steps on the original dataset, from generating base clusters to consensus formation. Results from different consensus clustering can be merged, considering a user-specified weighting. Visual comparisons can be performed between the number of clustersMain functions: cluscomp(), clrob(), memrob()Methods for sampling: Random sampling of observations and variation of parameters/algorithmsMethods for base clusters: Hierarchical clustering, divisive analysis, partitioning around medoids, and k-means, as well as custom-defined clustering algorithmsEnsemble methods: Hierarchical consensus clustering based on the method developed by Monti et al. (2003)(Simpson, 2022; Simpson et al., 2010)M3C package uses Monte Carlo simulation to simulate stability scores distribution, in order to inform the selection of optimal number of clusters (via hypothesis testing)Main functions: M3C()Methods for sampling: Random sampling of observationsMethods for base clusters: Partitioning around medoids, k-means, spectral clustering, hierarchical clusteringEnsemble methods: Monte Carlo reference-based consensus clustering based on Monti et al. (2003), regularised consensus clustering (addition of penalty term to the estimation of the optimal K). This model was developed to address the issue of existing methods (e.g., PAC) in identifying optimal clusters, which allows comparisons with null distributions using hypothesis testing(John & Watson, 2023; John et al., 2020)cola package internally using clue for assigning consensus labels. Incorporates top-values methods to select variables with highest variability as part of the pre-processing stage. It can also perform hypothesis testing to identify specificity of variables in one or several clusters (e.g., using permutation)Main functions: consensus_partition(), run_all_consensus_partition_methods()Methods for sampling: Random sampling of observations or variables and variation of algorithmsMethods for base clusters: Hierarchical clustering, k-means, spherical k-means, partitioning around medoids, and model-based clustering, as well as custom-defined clustering algorithmsEnsemble methods: Internal use of cl_consensus() from the clue package(Gu, 2023; Gu et al., 2021)Other related packages:coca: an extension of ConsensusClusterPlus, which focuses on multi-view clustering, using a multi-cluster-of-clusters analysis (COCA). The individual results from individual views were combined using an unweighted approach(Cabassi & Kirk, 2020a, 2020b)klic: multi-view clustering using the kernel learning integrative clustering (KLIC) method, which is similar to coca. klic uses the kernel k-means approach, which allows one to assign weights to different views, aimed at removing view-specific noise(Cabassi & Kirk, 2020b; Cabassi et al., 2020)SC3: specific consensus package for clustering single-cell RNA-Seq data. The consensus clustering approach is similar to the method proposed by Monti et al. (2003) except that BCs were generated from data with different methods of distance calculation and transformation(Kiselev, 2023; Kiselev et al., 2017)clusterExperiment: specific consensus package for clustering single-cell and other large gene expression data(Purdom & Risso, 2023; Risso et al., 2018)BCClong: Bayesian consensus clustering for multi-view longitudinal data using a joint modelling approach(Tan et al., 2022, 2023)Not all packages are in CRAN, and some packages will not be maintained regularly. Please check detailed package documentation when using these packages Ensemble clustering implementation in R diceR can perform all the steps on the original dataset, from generating base clusters to consensus formation. It includes a large number of methods for base clustering as well as methods for visualisation of results that inform hypothesis testing Main functions: dice(), consensus_cluster(), consensus_matrix(), consensus_combine(), consensus_evaluate() Methods for sampling: Random sampling of observations Methods for base clusters: Nonnegative matrix factorisation, hierarchical clustering, divisive analysis clustering, k-means, partition around medoids, affinity propagation, spectral clustering, Gaussian mixture model, co-clustering using multiple latent block model, self-organising map with hierarchical clustering, fuzzy c-means clustering, and hierarchical DBSCAN, as well as custom-defined clustering algorithms Ensemble methods: Majority voting, k-modes (categorical variable clustering of cluster label), cluster-based similarity partitioning algorithm (CSPA, graph partitioning of co-association matrix), link-based cluster ensembles (LCE, spectral graph partitioning of co-association matrix), and LCA (categorical variable clustering of cluster label). In the diceR package, CSPA and LCE applied hierarchical clustering instead of the original graph models proposed by Strehl and Ghosh (2002) and Iam-On and Garrett (2010) ConsensusClusterPlus is a simple package based on two functions only, one generating the base clusters and implementing consensus clustering, and the other calculating cluster-consensus and item-consensus. Several graphical features are incorporated in these two functions Main functions: calcICL(), ConsensusClusterPlus() Methods for sampling: Random sampling of observations and/or variables Methods for base clusters: Hierarchical clustering, partitioning around medoids, and k-means, as well as custom-defined clustering algorithms Ensemble methods: Hierarchical consensus clustering based on the method developed by Monti et al. (2003), i.e., hierarchical clustering based on co-association matrix clue forms consensus on a collection of partitions (soft or hard) or hierarchies (n-valued trees) by minimisation of dissimilarity between consensus candidates and base clusters (“optimisation approach”) Main functions: cl_consensus(), cl_dissimilarity(), cl_ensemble() Methods for sampling: Random sampling of observations and variation of parameters/algorithms Methods for base clusters: Partitions or hierarchies must be created prior to using the package with the desired algorithm. Both soft/fuzzy and hard/crisp partitions are possible Ensemble methods: Minimisation is done differently for partitions and hierarchies, with a range of possible methods. For partitions: fixed-point algorithm for soft/hard least squares Euclidian/median Manhattan consensus partitions, fixed-point algorithm for the “first model” from Gordon and Vichi (2001), greedy algorithm of Dimitriadou et al. (2002), sequential unconstrained minimisation techniques. For hierarchies: minimisation of Euclidean dissimilarity, sequential unconstrained minimisation technique with Manhattan dissimilarity, majority consensus tree of Margush and McMorris (1981). Custom-defined algorithms can be used for partitions and hierarchies clusterCons package performs all the steps on the original dataset, from generating base clusters to consensus formation. Results from different consensus clustering can be merged, considering a user-specified weighting. Visual comparisons can be performed between the number of clusters Main functions: cluscomp(), clrob(), memrob() Methods for sampling: Random sampling of observations and variation of parameters/algorithms Methods for base clusters: Hierarchical clustering, divisive analysis, partitioning around medoids, and k-means, as well as custom-defined clustering algorithms Ensemble methods: Hierarchical consensus clustering based on the method developed by Monti et al. (2003) M3C package uses Monte Carlo simulation to simulate stability scores distribution, in order to inform the selection of optimal number of clusters (via hypothesis testing) Main functions: M3C() Methods for sampling: Random sampling of observations Methods for base clusters: Partitioning around medoids, k-means, spectral clustering, hierarchical clustering Ensemble methods: Monte Carlo reference-based consensus clustering based on Monti et al. (2003), regularised consensus clustering (addition of penalty term to the estimation of the optimal K). This model was developed to address the issue of existing methods (e.g., PAC) in identifying optimal clusters, which allows comparisons with null distributions using hypothesis testing cola package internally using clue for assigning consensus labels. Incorporates top-values methods to select variables with highest variability as part of the pre-processing stage. It can also perform hypothesis testing to identify specificity of variables in one or several clusters (e.g., using permutation) Main functions: consensus_partition(), run_all_consensus_partition_methods() Methods for sampling: Random sampling of observations or variables and variation of algorithms Methods for base clusters: Hierarchical clustering, k-means, spherical k-means, partitioning around medoids, and model-based clustering, as well as custom-defined clustering algorithms Ensemble methods: Internal use of cl_consensus() from the clue package Not all packages are in CRAN, and some packages will not be maintained regularly. Please check detailed package documentation when using these packages Example of algorithm implementation Dataset For simplicity in demonstrating different algorithms, we have used a simulated dataset generated from multivariate normal distributions. This simplified example does not capture the complexity of real-world data structures and challenges (e.g., sparsity, skewed distributions, missing data, or high dimensionality). However, this relatively simple generative process enables readers to become familiar with the analytical procedures and underlying methodological concepts. The code for generating the data is provided in Supplementary Material II. Briefly, we applied three steps: (1) generated a mixture of three multivariate normal distributions (with different means and variances) on a three-dimensional space representing three clusters with n = 300, 150, and 150; (2) randomly projected the original three-dimensional data onto a 10-dimensional space, creating a dataset with 600 observations and 12 variables; and (3) added random errors to individual variables and additional outliers. The distributions of the three clusters are visualised using principal component analysis (PCA, see Fig. 6).Fig. 6PCA of the three generated clusters in the simulated example dataset PCA of the three generated clusters in the simulated example dataset Steps of ensemble clustering In this section, the detailed procedures for implementing ensemble clustering models are presented. The R code for the step-by-step guide of modelling processes can be found in Supplementary Material III. Developing the analysis plan The development of an analysis plan before obtaining or analysing the data can be difficult for clustering analysis, as the choice of the appropriate model may depend on the characteristics of the data and the performance of different algorithms. However, it remains important to develop the framework for the analytical procedures and guidance for practical decisions. Ideally, this process should be pre-registered to support robust open science practices (Gao et al., 2023a, 2023b). A template analysis plan with the current tutorial analysis as an example is provided in Table S2 in Supplementary Material I. Data assessment and pre-processing Researchers should undertake in-depth examination of source data for characteristics such as dimensionality, distribution of variables, and level of missing data, which can be done using functions such as dfSummary from the summarytools package. This process should be followed by data pre-processing procedures (e.g., identification of outliers, normalisation, and dimensionality reduction, see Supplementary Material III). In this example, we employ principal component analysis (PCA) for dimensionality reduction. Many other dimensionality reduction methods can be used such as multidimensional scaling (MDS), t-distributed stochastic neighbour embedding (t-SNE), nonnegative matrix factorisation (NMF), and uniform manifold approximation and projection (UMAP); see detailed comparisons of these models elsewhere (Ayesha et al., 2020; Jia et al., 2022; Van Der Maaten et al., 2009). NMF and related methods have been widely applied in the ensemble clustering framework, notably in the fields of genetics and neuroscience, to effectively manage high-dimensional and sparse data (Brunet et al., 2004; Gaujoux & Seoighe, 2010). Ensemble clustering Many existing packages provide individual functions for each step of e clustering (generating BCs and ensemble BCs). As methods involving random sampling of observations will introduce missing data, one additional imputation step is also required. The quality and diversity of BCs can also be examined. Poor-quality BCs (lower ranking across a range of internal validity indices) can be trimmed or removed. The detailed code for carrying out the two ensemble clustering steps is provided in Supplementary Material II using the diceR package. However, most users may choose to use the integrated approach with one function incorporating both steps. Using the dice function from the diceR package, this can be achieved using the following code; additional specifications (e.g., trim, reweight, and plot) can be found in the user manual (https://cran.r-project.org/web/packages/diceR/refman/diceR.html#dice). In this analysis, we created four solutions with cluster numbers varying from 2 to 5, utilising three BC algorithms, k-means ("km"), Gaussian mixture model (GMM, "gmm"), and hierarchical clustering ("hc") with Ward’s linkage method (hc.method = "ward.D2"), along with one consensus function, LCE (cons.funs = "LCE"). dice offers a few consensus functions, such as k-mode and majority voting, among which CSPA is consistent with the model specification proposed by Monti et al. (2003). The function is also set up to automatically calculate internal validity indices, which facilitate the evaluation of BCs, allowing for the automatic exclusion of those with low scores on internal indices (by setting trim = TRUE), and the reweighting of BCs based on their internal validity (by setting reweigh = TRUE). Specifying plot = TRUE will allow us to generate associated plots (e.g., consensus matrix, see Fig. 6, with interpretation provided in subsequent sections). Similar code can also be achieved using the ConsensusClusterPlus function from the ConsensusClusterPlus package. In this package, we can perform both random sampling of variables and observations, in contrast to diceR, which only allows random sampling of observations. Users can also manually specify the two steps with the support of existing packages (e.g., to incorporate imputed datasets currently not directly embedded in packages such as diceR); see the example provided in Supplementary Material III. Specifying and parameterising ensemble clustering models Ensemble clustering models offer a variety of specification options and parameter settings, such as the selection of BC algorithms, consensus matrix types, the number of BCs, and the fraction of samples to utilise. Typically, a sampling proportion of 80% is recommended (Monti et al., 2003; Șenbabaoğlu et al., 2014) to capture most data characteristics while allowing for variations among BCs. In this illustrative example, we use 100 BCs. In practice, the ensemble size can be determined by how diverse BCs are, as highlighted above. Users also have the option to explore how the results change with an increasing number of BCs. Choosing the optimal parameters and accessing validity A critical and primary benefit of ensemble clustering is that the level of consensus between BCs can also be used to choose the optimal number of clusters based on stability, as discussed above. As shown in Fig. 6, when the number of clusters (k=3k=3) was chosen based on the ground truth in the simulated data, the consensus matrix showed three clear square-shaped structures (with distinct boundaries) along the antidiagonal direction, suggesting that pairs of observations were commonly clustered into the same group (or different groups). When other numbers of clusters were chosen, there were no clear antidiagonal structures. To quantify this visual dispersion, Brunet et al. (2004) applied a measure based on the cophenetic correlation coefficient, which measures the correlation between the consensus co-association matrix and the distance on the reordered co-association matrix in Fig. 7.Fig. 7Consensus matrix constructed from 100 BCs estimated with GMM, with the number of clusters (k) ranging from 2 to 5, using the diceR function in the diceR package with the simulated example dataset Consensus matrix constructed from 100 BCs estimated with GMM, with the number of clusters (k) ranging from 2 to 5, using the diceR function in the diceR package with the simulated example dataset Monti et al. (2003) further developed a methodology to facilitate choosing kk using the distribution of consensus matrix index. An optimal number of kk will commonly produce a consensus matrix index with a bimodal distribution, with a high number of indexes closer to 0 (always clustered to different groups) or 1 (always clustered to the same group). This distribution can be described using an empirical cumulative distribution function (CDF) and assessed by considering changes in the area under the CDF curve, known as the delta area (see Fig. 7). As suggested by Monti et al. (2003), the CDF derived from the optimal kk typically exhibits three characteristics: a three-phase step function in the CDF plot, beginning with an initial step near 0 (indicative of the proportion of zeros in the matrix), followed by a flat line across the 0–1 range, and concluding with a final step near 1. This is reflected as an “elbow” effect seen in the relative change in the area under the delta area plot; see Fig. 8 (k=3k=3 for all three BC algorithms). The CDF method can sometimes identify clusters in data where none exist (see examples provided by Șenbabaoğlu et al., 2014). To address, this issue, the authors introduced the proportion of ambiguous clustering (PAC) score (Șenbabaoğlu et al., 2014). This metric measures the proportion of sample pairs whose consensus indices fall within an indeterminate range (e.g., 0.05–0.95, which can be tuned) indicating observations that are more ambiguous in cluster membership. A lower PAC score indicates a flatter middle segment in the CDF, reflecting greater consistency across BCs. PAC scores are computed in the dice function, which also serves as the default method for selecting the optimal kk (when k.method = “all” is not specified). However, PAC was also suggested to have an inherent bias towards higher values of kk because the null reference distributions are not considered (John et al., 2020).Fig. 8Cumulative distribution function (CDF) of consensus matrix indexes (A) and delta area plot representing changes in CDF (B) estimated from the diceR function in the diceR package using the simulated example dataset Cumulative distribution function (CDF) of consensus matrix indexes (A) and delta area plot representing changes in CDF (B) estimated from the diceR function in the diceR package using the simulated example dataset Apart from the number of clusters, an ensemble clustering pipeline also involves many other hyper‑parameters that must be set before the algorithm can be run—for example, the dimensionality retained, the covariance structure chosen on finite mixture models, the linkage criterion used in hierarchical agglomeration, and the neighbourhood radius in density‑based methods. In supervised prediction problems, hyper‑parameters are typically tuned by maximising performance on unseen test set(s) (e.g., by cross‑validation). However, as clustering is unsupervised, no direct signal is available; consequently, hyper‑parameters must be chosen on the basis of indirect evidence such as internal, external, predictive, and translational validity of the resulting groups (see explanation in Table S2). Internal validity can be evaluated using CV or nested CV framework to ensure that the decision-making process is not impacted by random fluctuations in the data. This can be evaluated using multiple stability indicators, including PAC and CDF, as well as other indicators such as the adjusted Rand index (Gao et al., 2023a, 2023b). This approach is detailed in the examples provided by Dwyer et al. (2022). One operational example is provided in Fig. S3 and Supplementary Material IV. To confirm that the cluster solution did not occur by chance, permutation-based non-parametric tests can be incorporated (Dwyer et al., 2022). In a permutation test, an empirical distribution of a statistic under the null hypothesis is constructed by reshuffling the observed data many times and recalculating the statistic of interest for each reshuffled dataset. The statistic calculated for the ensemble cluster is then compared with this empirical distribution to obtain a p-value. An example of such a test can be found in Supplementary Material III, where the PAC score is calculated for 100 permutations of the BC classifications obtained through k-means, and compared with the PAC score of the original results. The p-value obtained was 0, irrespective of the bounds selected for the PAC score, due to the clear superiority of k-means clustering over random assignment of observations to clusters. Assessing the external validity, that is, the generalisability of the clustering results to external data (data from other settings, populations, or time periods), remains a significant challenge, primarily due to the absence of ground truth. While various methods have been proposed, no consensus on the processes and procedures for such validation has been established. This situation is largely attributed to the perception of clustering analysis as an exploratory technique (Ullmann et al., 2022). However, we believe that external validity is desirable to allow findings to be translated into practice. We acknowledge that not all studies can find a matched external validation dataset. In these cases, researchers can choose to publish a prediction algorithm (algorithm predicting cluster labels), enabling others to validate when such data are available. When the external validation data are available, the validation can either carry out two independent clustering analyses and directly compare the similarity of clustering results or apply a predictive algorithm (Brucar et al., 2023). For example, Ullmann et al. (2022) proposed an approach that compares multiple indexes and performance indicators across models, replicating the clustering findings (rerun of the identical model on validation data) or predicting clustering results (cluster labels in the validation data obtained from prediction models of clustering labels). When the two validation approaches yield comparable clusters (e.g., showing similar relationships with the input data), consistent correlations with external factors, and concordance between the two sets of results, this indicates a high level of external validity for the findings. External variables not included in the clustering process (e.g., sociodemographic factors and other associated outcomes such as diagnoses and long-term trajectories) can be used to assess the predictive and translational validity (Brucar et al., 2023). Reporting ensemble clustering studies Studies employing ensemble clustering should follow the reporting guidelines of clustering studies (Gao et al., 2023a, 2023b). In addition, users should include detailed modelling procedures (e.g., BC algorithms, consensus function, the method for choosing optimal kk) and choice of parameters (e.g., sampling proportion, number of BCs) as well as associated results (e.g., consensus matrix, CDF and Delta area plot, PAC scores). Important practical notes Importantly, limitations of BC algorithms will be automatically carried forward in the consensus process (e.g., assumptions in the shapes of clusters, sensitivity to scale, outliers, and density differences between clusters, impacts of hyperparameter choices); see details discussed elsewhere (Gao et al., 2023a, 2023b). Clusters with arbitrary or non-convex shapes may not be recovered by BC algorithms that are designed to identify primarily convex cluster structures. In this case, the clustering solution may appear to be stable but fail to recover the true underlying heterogeneity. Therefore, it is important to evaluate whether the assumptions of BC algorithms can be met and test the impact of choosing different BC and ensemble algorithms. Users can take advantage of the diceR package, which allows for simultaneously employing and comparing multiple algorithms (see details in Supplementary Material II). Stability and internal validity do not guarantee external validity; similarly, stability and internal and external validity do not translate directly to meaningfulness. Clustering solutions that are stable and describe the data well can fail to demonstrate external validity. Such failures may arise from many different factors, such as differences in the level of noise, underrepresented subgroups, variations in data collection and processing pipelines, and overfitting of the original clustering solutions (Wen et al., 2025). Conversely, stable, valid, and generalisable clustering solutions may nevertheless have very little practical utility. For example, the partition of “low”, “medium”, and “high” categories may provide little actionable insight. Therefore, the choice of modelling approaches and perimeters will need to be guided by the aim of the research and the potential implications of the findings, rather than purely relying on a data-driven approach. Comparison To demonstrate the need for ensemble clustering, we conducted simulations using datasets from three published studies (Amiet et al., 2020; Cotton et al., 2022; Jakobson & Rigby, 2021) (see details of these studies in Table S3 in Supplementary Material I). The primary aim of these simulations was to examine the extent to which ensemble clustering improves the stability and robustness of clustering solutions relative to single-run models when data are subject to small perturbations. Meaningful clustering solutions should ideally remain stable under minor changes in the data, with similar observations consistently grouped into the same clusters. We therefore evaluated clustering stability by introducing varying levels of perturbation to the data. For each simulation dataset, we generated perturbed datasets by randomly removing 0.25–20% of the data and filled the omitted data with multivariate imputation by chained equations (MICE). This process ensures that the replacement values follow the same multivariate distribution as the original data. For each perturbation condition (e.g., 0.25% missing), imputation was repeated 100 times to generate multiple perturbed datasets. We then applied each clustering algorithm repeatedly to the perturbed datasets within a perturbation condition (e.g., 0.25% missing), and the stability of a clustering algorithm was characterised by the Rand index (RI, which measures levels of agreement between pairs of clustering results). The larger RI values suggest that the algorithm can produce more stable and generalisable results when there are minor changes in the data. Detailed procedures and results are provided in Supplementary Material V. Results from three studies are provided in Fig. 9. Minor changes in data (e.g., 0.25% of the data being different) introduced substantially different results. In some cases (e.g., hierarchical clustering), stability was only slightly better than random sampling (see Fig. 9A). Ensemble clustering, on the other hand, had the best performance with the slowest decay in stability with increasing perturbation. This is particularly the case in the example in Fig. 9B, where the source data have slightly higher dimensionality. Single-run models also showed substantially wider error bars for the pairwise RI across three simulation studies, indicating greater variability in agreement across perturbed dataset pairs. This again highlights the lack of stability of single-run clustering solutions and their greater sensitivity to data perturbations than ensemble clustering. This simple example, although demonstrating relative robustness and reproducibility of ensemble clustering solutions under data perturbation, does not imply that its solutions accurately reflect the underlying heterogeneity.Fig. 9Changes in average clustering stability (measured using the pairwise Rand index, RI) across perturbed datasets (100 replications for each perturbation condition) using four types of clustering models (error bar: standard deviation). Note: The source datasets were obtained from A clustering of 20 AQoL-6D items (Cotton et al., 2022) with four clusters identified; B clustering of alexithymic traits (Jakobson & Rigby, 2021) with two clusters identified; and C clustering of cannabis expectancies (Amiet et al., 2020). In this demonstration, RI is chosen as the stability index due to its intuitive interpretation. However, RI is sensitive to the specified number of clusters (kk). Therefore, we use a dashed line to represent the RI expected under random cluster assignment while preserving the number of clusters identified in the original study, providing a theoretical lower-bound (chance-level) benchmark for clustering agreement Changes in average clustering stability (measured using the pairwise Rand index, RI) across perturbed datasets (100 replications for each perturbation condition) using four types of clustering models (error bar: standard deviation). Note: The source datasets were obtained from A clustering of 20 AQoL-6D items (Cotton et al., 2022) with four clusters identified; B clustering of alexithymic traits (Jakobson & Rigby, 2021) with two clusters identified; and C clustering of cannabis expectancies (Amiet et al., 2020). In this demonstration, RI is chosen as the stability index due to its intuitive interpretation. However, RI is sensitive to the specified number of clusters (kk). Therefore, we use a dashed line to represent the RI expected under random cluster assignment while preserving the number of clusters identified in the original study, providing a theoretical lower-bound (chance-level) benchmark for clustering agreement Discussion Clustering techniques designed in the twentieth century have contributed important insights in psychology and psychiatry; however, challenges related to reproducibility, stability, and clinical translation have limited their real-world impact. In this paper, we aimed to address this problem by translating ensemble clustering techniques to audiences in the field. As outlined by our tutorial, ensemble clustering is a flexible technique that can be relatively easy to implement with existing toolboxes to address related questions. We then demonstrated that minor perturbations of psychological/psychiatry data can lead to major differences in clustering solutions using traditional single-run methods, which are reduced with ensemble clustering. When combined, future research in the field will benefit from these techniques, and we advocate their widespread use. In addition to a simulated demonstration example that serves as a step-by-step guide for the ensemble clustering technique, we provided a comparative analysis with different clustering methods and data missing rates to evaluate the performance of ensemble clustering in three complex real-world scenarios in psychology and psychiatry. The results demonstrated the significant reliability and application value of ensemble clustering technology in these fields. The resources presented will enable researchers to apply advanced ensemble clustering techniques to decompose heterogeneous and complex psychological data into stable subgroups. Ensemble clustering, along with its extensions such as multi-view clustering (see one of the examples provided by Pfeifer et al., 2023), offers a robust, data-driven framework that enables the integration of high-dimensional, multimodal data and diverse clustering paradigms (Dwyer et al., 2022; Gao et al., 2023a, 2023b). These approaches support insights such as the understanding of the heterogeneous biological, psychological, and social mechanisms of mental health conditions and precision medicine approaches to treatment (Dwyer et al., 2018; Manchia et al., 2020; Wright & Woods, 2020). Additionally, ensemble clustering can increase confidence in clustering results by improving the stability and robustness of identified subgroups, thereby supporting more reliable interpretation and translation of the knowledge into real-world practice. Psychological and psychiatric data present several methodological challenges that warrant consideration when applying ensemble clustering approaches. These data frequently comprise ordinal (e.g., Likert-scale) variables, latent constructs, and highly correlated variables. Treating highly corelated ordinal items, underpinned by latent constructs, as continuous variables and directly integrating them into distance-based clustering models will be problematic. These features can distort the distance matrix, producing stable clustering solutions that are a poor fit for the data (e.g., producing cluster results dominated by one overly represented domain among source variables). In these cases, the application of proper BC models (e.g., model-based clustering) or dimensionality reduction methods is necessary. Longitudinal data, with trajectory changes and dynamic relationships between variables, can introduce further complexity. The typical ensemble approach cannot incorporate the full features of advanced longitudinal models, such as growth mixture models, which model latent subgroups of longitudinal trajectory random effects within subgroup. In these cases, assembly is difficult, as disagreement between models may arise from many different parameters (e.g., trajectory shapes, variance structures), making it unclear which model components should be meaningfully combined into a consensus solution. In such cases, different ensemble pipelines are required, e.g., via Bayesian models (Xiao et al., 2025). It is important to note that ensemble clustering techniques also come with several limitations. First, they demand considerable computational resources, especially when processing large datasets. This can be a major barrier for applying ensemble clustering models, which can be addressed by applying computationally efficient models (e.g., using a binary cluster association matrix) and implementing computation pipelines in high-performance computing clusters. Second, the flexibility in choosing different clustering methods, optimisations, and strategies may introduce additional complexities. These choices are often important because the limitations of the underlying BC algorithms can also be carried forward in the consensus approach. Despite the substantial advantages offered by ensemble clustering models, their deployment also necessitates validation in external samples. Further research and development will undoubtedly address some of these limitations, but these methods have already demonstrated their potential in other fields. Developing familiarity with the conceptual framework as well as practical skills for applying these models would be a great advantage for research in psychology and psychiatry. Supplementary Information Below is the link to the electronic supplementary material.ESM 1Supplementary Material 1 (DOCX 352 KB)ESM 2Supplementary Material 2 (PDF 295 KB)ESM 3Supplementary Material 3 (PDF 2.80 MB)ESM 4Supplementary Material 4 (PDF 1.33 MB)ESM 5Supplementary Material 5 (PDF 1.04 MB) Supplementary Material 1 (DOCX 352 KB) Supplementary Material 2 (PDF 295 KB) Supplementary Material 3 (PDF 2.80 MB) Supplementary Material 4 (PDF 1.33 MB) Supplementary Material 5 (PDF 1.04 MB)