On estimating the effective sample size of phylogenetic trees in an autocorrelated chain

Jonathan Klawitter ,⋆, Lars Berling, Jordan Douglas, Dong Xie, Alexei J. Drummond · 2026

Read On estimating the effective sample size of phylogenetic trees in an autocorrelated chain by Jonathan Klawitter ,⋆, Lars Berling, Jordan Douglas, Dong Xie, Alexei J. Drummond (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03521.txt.

Introduction

Jonathan Klawitter and four others ask how many truly separate trees a Bayesian evolution study has sampled. Such analyses explore evolutionary trees with Markov chain Monte Carlo, and consecutive samples resemble each other. The effective sample size corrects for that stickiness, but trees are discrete and high-dimensional, so usual estimators falter. Their 2026 study compares current tree gauges against new ones built on conditional clade distributions.

A further new gauge reads clade frequency differences between independent chains. Simulated chains with known answers grade every method, and real data sets test stability. The clade-distribution gauges match current approaches, and two of them vary less by averaging many estimates. The chance-based gauge and two previously recommended methods cost too much on long chains. Multimodality and poor mixing can badly distort all the gauges' readings, the authors warn.

Catalog record

CREATORJonathan Klawitter ,⋆, Lars Berling, Jordan Douglas, Dong Xie, Alexei J. Drummond
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE104533
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.03521

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On estimating the effective sample size of phylogenetic trees in an autocorrelated chain Jonathan Klawitter ,⋆, Lars Berling, Jordan Douglas, Dong Xie, Alexei J. Drummond Abstract Estimating the effective sample size (ESS) is fundamental in Bayesian phylogenetic inference to properly account for autocorrelation in MCMC samples. While methods for continuous parameters are well established, the discrete and high-dimensional nature of treespace poses substantial challenges. Here, we compare existing tree ESS estimators with novel approaches that leverage tractable tree distributions, specifically Conditional Clade Distributions (CCDs), as well as a new probabilistic estimator based on clade frequency differences between independent chains. Using simulated chains with known ESS bounds, we assess estimator accuracy and evaluate their stability and robustness on simulated and real datasets. We further examine how multimodality in posterior distributions and poor mixing can substantially affect ESS estimates, highlighting the need for careful interpretation. Our CCD-based estimators perform comparably to existing approaches, with two methods showing lower variance by averaging across multiple estimates. In contrast, the probabilistic estimator and two previously recommended methods incur prohibitive computational costs for long chains. Together, these results provide guidance for reliable and efficient tree ESS estimation in complex phylogenetic analyses. 1 Introduction Bayesian inference with a Markov chain Monte Carlo (MCMC) algorithm is one of the main inference paradigms in phylogenetics [ 39 , 24 , 42 , 6 ] . MCMC algorithms explore the parameter space by proposing changes to the current state (such as tree topology modifications or parameter value adjustments) and accepting or rejecting these proposals based on their likelihood and prior probability. The objective is to construct a Markov chain that effectively converges to the posterior distribution of phylogenetic trees along with associated parameters, such as substitution and clock rates. This posterior distribution spans the entire space of possible trees (the treespace ) and their parameters. In practice, however, we can only approximate this posterior through a finite sequence of samples collected during the MCMC run. A key challenge of MCMC is autocorrelation: consecutive samples are typically dependent, which reduces the effective sample size (ESS) and lowers the precision of posterior estimates [ 14 , 38 , 47 ] . This dependence is quantified by the autocorrelation time (ACT) , which measures the average number of iterations needed to obtain effectively independent samples. For a chain of length N , the ESS ( N_{\textup{Eff}} ) is inversely related to the ACT ( \tau ) by the equation N_{\textup{Eff}}=N/\tau . Thus, a higher ACT directly translates to a lower ESS, requiring more iterations to achieve reliable posterior estimates. This challenge is particularly acute in phylogenetics, where MCMC must navigate the complex, high-dimensional, and non-Euclidean phylogenetic treespace using local tree rearrangement operations [ 1 , 5 , 7 ] . Understanding the ESS is crucial for properly accounting for autocorrelation in downstream analyses and assessing the quality of posterior estimates. The ESS quantifies how many independent samples would provide the same statistical precision as the autocorrelated MCMC samples. A low ESS relative to the total number of draws indicates that the chain is moving slowly through the parameter space, signalling that more samples are needed for accurate estimates. See Magee et al. [ 35 ] for an in-depth study of the relation between the ESS of phylogenetic trees and the Monte Carlo error. Monitoring ESS is therefore essential for assessing MCMC efficiency and determining whether sufficient effectively independent samples have been collected. Since MCMC in phylogenetics explores multiple parameter spaces with varying efficiency, the ESS must be calculated separately for each parameter. In the field of phylogenetic MCMC, a common rule of thumb suggests a minimum ESS of 200 for all important parameters, including the posterior density and likelihood. Magee et al. [ 35 ] and Fabreti and Höhna [ 16 ] advocate for higher thresholds of 500 and 625, respectively. Reliable ESS estimation provides several critical benefits for phylogenetic MCMC analyses. First, a low ESS often indicates that an MCMC chain should be run for additional iterations. While a large ESS does not guarantee convergence, it serves as a valuable diagnostic for assessing convergence when used alongside other lines of evidence, as discussed by Berling, Bouckaert, and Gavryushkin [ 3 ] . Reliable ESS estimation is thus an essential post-processing step for interpreting and ensuring the robustness of MCMC results. Second, ESS and ACT values quantify sampling efficiency, enabling researchers to compare different MCMC algorithms, evaluate the effectiveness of proposal mechanisms, and optimize tuning parameters. Third, the ESS is fundamental for computing Monte Carlo standard errors, which quantify the precision of posterior estimates due to finite sampling [ 16 , 35 ] . Finally, in Bayesian model averaging [ 48 ] and model selection approaches such as AIC and BIC, the ESS can be used to adjust effective sample sizes or serve as weights to prevent overestimation of model support [ 26 ] . We tackle the challenging problem of estimating the ESS for samples of phylogenetic tree topologies (i.e., without branch lengths). The fundamental difficulty stems from the discrete, high-dimensional nature of treespace, which lacks the metric properties of Euclidean spaces and therefore cannot accommodate existing ESS estimation methods designed for continuous parameters. Related Work. To estimate the ESS for samples of phylogenetic tree topologies, previous approaches have mapped the sampled trees to traces of real-valued parameters, enabling the application of existing ESS estimators developed for continuous variables [ 14 ] . Lanfear, Hua, and Warren [ 32 ] introduced the tree ESS estimator pseudoESS , which computes the distance from each sampled tree to a set of randomly selected reference (focal) trees from the sample, derives ESS estimates from these distance traces, and reports the median value along with summary statistics. They proposed plotting these distances as tree topology traces and in addition also propose jump distances which visualize tree distances between pairs of trees at increasing sampling intervals. In a related vein, the RNNI distance [ 7 ] to a fixed tree estimate (UPGMA and neighbour joining) is used to address gene tree topology convergence in the StarBeast3 multispecies coalescent program [ 9 ] . Fabreti and Höhna [ 16 ] took a different approach, converting (unrooted) trees into binary traces that indicate the presence or absence of specific splits (or clades for rooted trees). Magee et al. [ 35 ] adapted the initial positive sequence estimation method—a popular ESS estimator for Euclidean variables—for use with trees by approximating the covariance structure through a tree metric. They named their method frechetCorrelationESS to reflect its use of the Fréchet mean and variance. Lanfear et al. [ 32 ] also proposed a similar approach, approximateESS , which employs a more heuristic approach to estimating ACT and ESS. In a comparative study, Magee et al. [ 35 ] evaluated the accuracy of these and additional ESS estimators using simulated chains. Their evaluation was based on comparing estimated values to known Monte Carlo sampling errors, using simulated chains that replicate key characteristics of real phylogenetic posterior tree samples. Contribution. In this paper, we present new methods for estimating the effective sample size of phylogenetic tree topologies, a refined evaluation framework based on simulated chains with known ESS bounds, and a case study examining

Read the complete text