Comparing automated subcortical volume estimation methods; amygdala volumes estimated by FSL and FreeSurfer have poor consistency Authors: Patrick Sadil, Martin A Lindquist Published: 2024 ABSTRACT AbstractSubcortical volumes are a promising source of biomarkers and features in biosignatures, and automated methods facilitate extracting them in large, phenotypically rich datasets. However, while extensive research has verified that the automated methods produce volumes that are similar to those generated by expert annotation; the consistency of methods with each other is understudied. Using data from the UK Biobank, we compare the estimates of subcortical volumes produced by two popular software suites: FSL and FreeSurfer. Although most subcortical volumes exhibit good to excellent consistency across the methods, the tools produce diverging estimates of amygdalar volume. Through simulation, we show that this poor consistency can lead to conflicting results, where one but not the other tool suggests statistical significance, or where both tools suggest a significant relationship but in opposite directions. Considering these issues, we discuss several ways in which care should be taken when reporting on relationships involving amygdalar volume.Keywords: amygdala, MRI INTRODUCTION Regional volumes of subcortex have been proposed as biomarkers for several psychopathologies. For example, the volume of the amygdala has been suggested as a biomarker for Alzheimer, depression symptom severity in young adults, bipolar disorder in youth, migraine frequency, chronic pain, and others (Daftary et al., 2019; Khatri & Kwon, 2022; Liu et al., 2017; Pfeifer et al., 2008; Rogers et al., 2009; Ruocco et al., 2012; Szeszko et al., 2004; Vachon‐Presseau et al., 2016). As a biomarker, subcortical volumes are advantageous for being interpretable (given the rich literature linking these structures to many functions), explainable (hypotrophy and hypertrophy are both easily described to healthcare providers and patients), and readily available. The latter point comes from the fact that it is possible to estimate the regional volumes from any structural image with several automated algorithms. For estimating regional subcortical volumes, two automated techniques are popular: FMRIB's Integrated Registration and Segmentation Tool (FIRST) from the FMRIB Software Library (FSL) and FreeSurfer's Automated Segmentation (ASEG) (Fischl, 2012; Patenaude, 2007; Patenaude et al., 2011). Both techniques exhibit high consistency with the gold standard of manual segmentation in healthy adults and some clinical populations (Dewey et al., 2010; Doring et al., 2011; Hsu et al., 2002; Lehmann et al., 2010; Morey et al., 2009; Nugent et al., 2013; Pardoe et al., 2009; Tae et al., 2008; Wenger et al., 2014), although there is variability across regions and between methods. For segmenting the hippocampus, FreeSurfer has been reported as having higher intraclass correlations than FSL (Doring et al., 2011), and neither method appears to have worse reliability (across repeated scans) than manual segmentation (Mulder et al., 2014). For the putamen, FSL has a higher Dice coefficient with manual segmentation, and the methods perform similarly on the caudate (Perlaki et al., 2017). For the amygdala, which method performs better depends on the metric (Morey et al., 2009). However, these comparisons may not generalize to other populations (e.g., pediatric, elderly), given that performance of the automated techniques is not consistently high across populations (Schoemaker et al., 2016; Kim et al., 2012; Sánchez‐Benavides et al., 2010; Zhou et al., 2021). Both tools may remain popular because their performance against a gold standard depends on the research context, with different situations favoring different tools. Given that both tools are often reasonable choices for a given study, and that the literature contains many reports that are based on one but not the other, we sought to understand how the tools compare with each other. To our knowledge, the two tools have only been compared with each other by Perlaki et al. (2017). In their research, the tools were consistent with each other (exhibiting intraclass correlations that ranged from around 0.7–0.9), but the analyses included only the putamen and caudate, and the sample size was relatively small (N = 30). That is, it remains unclear how often the two tools provide the same results and which factors affect these discrepancies. We extend the results of Perlaki et al. (2017) to the remaining subcortical structures using a much larger population (tens of thousands). Our investigation should be considered in the context of research that uses estimates of volume as a biomarker by, for example, correlating it with a health‐related outcome. Our primary concern is whether the results of such a study are expected to depend on the method used for automated segmentation. METHODS AND RESULTS First, we looked at the consistency of subcortical volumes between FSL and FreeSurfer using data from the UK Biobank (Alfaro‐Almagro et al., 2018). The data were downloaded Jan 2024 and contained 45,743 participants with usable anatomical data (Category 190: FreeSurfer ASEG; Category 1102: FSL FIRST). Agreement and consistency of the tools were measured using the single‐measurement intraclass correlation coefficient. For details on intraclass correlation calculations, see Appendix A1. Estimates and associated uncertainty are displayed using subscripts, as recommended by Louis and Zeger (2008). For example, an estimate of 0.22 with a 95% confidence interval spanning [0.21, 0.23] will be rendered as . Across all structures, estimated agreement was lower than estimated consistency (Table A1), reflecting differences in the average volumes estimated by the two methods (Appendix A2). However, a constant shift across participants would not affect many analyses targeted by our primary concern, analyses related to estimated correlations between regional volumes and some other measures. For that reason, we focus not on agreement but instead on consistency. Consistency varies by region (Figure 1, Table A1). To interpret the intraclass correlations, consider the categories provided by Cicchetti and Sparrow (1981): <0.4: poor, [0.4, 0.6): fair, [0.6, 0.75): good, [0.75, 1): excellent. Using those categories, the methods exhibit “good” to “excellent” consistency for most regions. However, the consistency of volumes for the amygdala is markedly worse than the others, being around only and for the left and right hemispheres (ranges of uncertainty span 95% confidence intervals). Compare those values to the values for the hippocampus (Figure 1), which has good consistency (left: , right: ). For both structures (i.e., the amygdala and hippocampus), consistency across hemispheres as reported by FreeSurfer is the highest numerically (Figure 1). Comparisons of subcortical volumes estimated by FSL and FreeSurfer for two example regions (a) hippocampus and (b) amygdala. For the remaining subcortical regions, see Table A1. In the lower triangular panels, the line of equivalence is marked with a solid line, and the dashed lines show the result of orthogonal regression. In the upper panel, the uncertainty estimates span 95% confidence intervals. The histograms along the diagonal display volumes in the full sample. Although our focus is on the consistency between tools, we note that their lack of agreement may lead researchers to make conclusions about whether one hemisphere tends to have a larger amygdala than the other. There is an interaction between method and hemisphere (FSL–FreeSurfer: 0.22, p < .001), with FreeSurfer reporting that the right amygdala is larger than the left (left–right: −0.19, p < .001) and FSL reporting that the left is larger than the right (left–right: 0.04, p < .001). As the effect side is almost zero in the latter, practically the interaction is driven by the right side. The amygdala has been described as particularly challenging to segment; one small study (N = 23) reports a consistency of 0.6 for volumes estimated by FSL across repeated scans of the same individual (Morey et al., 2010). Moreover, automated segmentation algorithms can be affected by experimental factors like site, scanner, participant positioning, and software version (Du et al., 2021; Hedges et al., 2022; Liu et al., 2020; McGuire et al., 2017; Morey et al., 2010; Mulder et al., 2014; Perlaki et al., 2017; Yang et al., 2016), and differential sensitivity to such factors could impact an intraclass correlation. In the UKB, several potentially confounding factors were correlated with estimates of amygdalar volume (Figure A2). However, regressing these factors from the estimates of volume did not improve the consistency between the methods (Appendix A4). With poor consistency between measurements of the amygdala, there is concern that reported relationships involving amygdala volumes may depend on which method is used for estimating the volume, a choice that may be considered arbitrary or lab‐specific. At least two kinds of issues could arise. First, lower consistency could make it more likely that one but not both methods lead to significant correlations. Second, lower consistency could make it more likely that the two methods produce significant correlations that go in opposite directions. To investigate how often these two issues could occur, we first simulated experiments with artificial data. In each simulation, datasets with two noisy estimates of volume and a third, outcome, variable were generated such that the two‐volume estimates had a pre‐specified intraclass correlation with each other (in expectation), and the true volume had a given product–moment correlation with the outcome variable. The estimated volumes were then tested for a product–moment correlation with the outcome, and the process was repeated for several intraclass correlations and sample sizes. In all simulations, the true product–moment correlation was set to a value that is either typical for neuroimaging research (0.1, Marek et al., 2022), small but non‐zero (0.01), or large (0.2). For additional details on the simulation methods, see Appendix A5. With lower consistency, the estimated product–moment correlations were more often on opposite sides of a significance threshold (Figure 2a, left column). For example, with consistency near the value that was estimated for volumes of the amygdala in the UKB (0.2), a typical effect size (0.1), and experiments with 50 participants, significance differed in around percent of simulations in which one test was significant (for simulations, estimates indicate medians and ranges of uncertainty span 95% equal‐tailed intervals, see Appendix A5). With a sample size of 100, that percentage was only minimally different (to ). For a discussion on this insensitivity to sample size, see Section Appendix A.6. Effects of low measurement consistency. In the subfigures, the left column or panel shows the proportion of simulated experiments where the methods produce correlations on opposite sides of an α=0.05 significance threshold. The right column or panel shows the proportion of simulated experiments where both measures are significantly correlated with an outcome but in opposite directions. Proportions are shown with color (Swihart et al., 2010). (a) Simulations with artificial data. Rows (rho) indicate pre‐specified product–moment correlation with the outcome variable. ICC: Pre‐specified intraclass correlation between measures of volume. (b) Simulations with UKB. Each row corresponds to a non‐image derived phenotype from the UKB. The value in parentheses is the absolute correlation of the variable with the average volumes of the left amygdala (average across methods). For a display without color that includes estimates of uncertainty, see Figure A3. Lower consistency also coincided with a higher proportion of experiments in which the two methods correlate in opposite directions (Figure 2a, right column). Considering the previous example (an intraclass correlation of 0.2, an effect size of 0.1, and 50 participants per experiment), around percent of experiments in which both product–moment correlations were significant resulted in the correlations having opposite signs. As the intraclass correlation increased, the rates of this effect decreased rapidly. To assess how often these two issues could occur in practice, we repeated the above analyses but with the UKB. From the UKB, we extracted the “Cognitive” variables using the FMRIB UKBiobank Normalisation, Parsing and Cleaning Kit (McCarthy, 2023), restricting analyses to variables with instance 2 and that exhibited one of the largest (in absolute magnitude) 50 rank correlations (between the cognitive variables and the average of the two estimates of the volume of the left amygdala), and further to only those participants that had values for all 50 of those variables (23,486 participants). Each simulation resulted in a rank correlation between the two volume estimates for the left amygdala and each of the 50 variables (results were comparable with the right amygdala). For additional details, see Appendix A5. Considering differing significance levels, the rates across experiments using the UKB resembled the rates from experiments with artificial data. In simulated experiments with 50 UKB participants in which at least one correlation was significant, one correlation was not significant in between and percent of simulations (the two estimates cover the 50 cognitive variables; Figure 2b, left panel). Proportions were generally lower for variables that were more strongly correlated with the measure of volume (for illustration, compare the variables that are higher versus lower in the left panel of Figure 2b). Considering significant correlations with differing signs, the rates across experiments with the UKB bracketed the rates with artificial data. In experiments simulated with 50 UKB participants, the rates ranged from to . For most variables, increasing the number of participants decreased the proportion of experiments in which the two correlations exhibited opposite signs, but for some the proportion increased. Across the 50 variables, 9 had a higher proportion at N = 100 than N = 50, 2 of which had non‐overlapping 95% equal‐tailed intervals (6348: duration to complete numeric path; 400: time to complete round of pairs matching game; see also Figure A3). DISCUSSION We examined the consistency of subcortical volumes within the UK Biobank (Alfaro‐Almagro et al., 2018), observing that two common methods of estimating the volume of the amygdala, one from FSL and one from FreeSurfer, have poor consistency with each other. We speculate that the lack of consistency for the amygdala compared with other subcortical structures is due in part to its relatively small size. For larger structures, discrepancies in a few voxels would have a smaller impact on consistency as compared with the impact of the same discrepancy on a smaller structure. When considering the difference in amygdala volume estimates between tools as a proportion of their average (Figure A1b), a trend is revealed where smaller structures tend to have more variable differences than larger ones. The amygdala is the smallest structure analyzed and has the largest variation. Even within other structures, the variability of the proportional difference increases as the average size decreases. Moreover, there is a clear correlation between head size and difference in amygdala volume (Figure A2), whereby smaller heads are associated with larger differences. Hence, smaller structures may be more difficult to segment consistently due to the proportional impact of discrepancies in the classification of a few voxels. The main concern in this report is that consistency this poor can lead to conflicting results. Two kinds of conflict were explored: the methods producing correlations that are on opposite sides of significance thresholds, and the methods producing volumes with significant correlations that have differing signs. The prevalence of these occurrences was estimated with artificial data and data from the UKB. Based on the observed rates, we make the following recommendations. Recommendations When testing for new biomarkers, report relationships with multiple automated methods (e.g., both FSL and FreeSurfer) Researchers may have idiosyncratic reasons for selecting a method, particularly when the choice is viewed as arbitrary. If the choice between methods is arbitrary, then reporting the outcome across selections clarifies the fragility or robustness of a result (for a general discussion, see Steegen et al., 2016). The choice may not always be arbitrary, as there are metrics along which and study populations for which one method may perform better (Huizinga et al., 2021; Morey et al., 2009; Zhou et al., 2021). Note that one effect of low consistency could be a downward bias on the magnitude of estimated correlations (Appendix A7), and so there may be advantages to not only reporting but also combining the estimates across methods when predicting health‐related outcomes (e.g., by averaging, or including both as independent variables in predictive models). Reporting estimates from multiple reasonable tools will help move conclusions beyond “there exists a correlation with the volume of the amygdala as estimated by method M (version x)” to simply “there exists a correlation with the volume of the amygdala.” When reviewing or conducting meta‐analyses of relationships with amygdala volume, consider the method that was used to estimate volume As mentioned in the Introduction, the amygdala has received substantial attention due to being predictive of an array of health‐related outcomes. As presented in this report, the volume that is estimated by one automated method may only weakly correspond to the volume estimated by another, and so it may be misleading to conduct a meta‐analysis without accounting for the algorithms used by the individual studies. In the UKB, there are differences in the strength of the correlations between the measures of volume and the cognitive variables; in nearly all of the variables considered in this report, the magnitude of the correlations involving FreeSurfer's method was numerically larger (Figure A5). Larger correlations do not imply higher veracity, but they further indicate that the methods track aspects of amygdalar anatomy differently. When replicating or extending research on a relationship that involves the volume of the amygdala, use the method reported in the original publications This recommendation follows standard practice for a replication study. We highlight it here in consideration of both extension studies that aim to apply a biomarker or biosignature that includes amygdala volume (such as when testing a putative relationship in a new population), and also in consideration of the ongoing evolution of methods for automatically estimating subcortical volumes. Although FSL and FreeSurfer are two of the most popular methods, others exist (e.g., Akhondi‐Asl & Warfield, 2013), including newer techniques based on deep‐learning approaches (e.g., Billot et al., 2023; for review, see Singh & Singh, 2021). Newer methods may have better correspondence with manual segmentation, warranting their use in replication or extension studies. But as this report shows, two methods can perform well while exhibiting poor consistency with each other. So when building on prior findings, it remains important to use the methods of those prior findings, even when they are superseded. AUTHOR CONTRIBUTIONS Patrick Sadil: Conceptualization; methodology; software; validation; formal analysis; investigation; resources; data curation; writing—original draft; writing—review and editing; visualization. Martin A. Lindquist: Conceptualization; methodology; validation; formal analysis; resources; writing—original draft; writing—review and editing; supervision; project administration; funding acquisition. CONFLICT OF INTEREST STATEMENT The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this article. Supporting information Data S1. Supporting Information. ACKNOWLEDGMENTS This work was supported by grant R01MH129397 from the National Institute of Mental Health. This research has been conducted using data from UK Biobank, a major biomedical database (Project ID: 33278). We are grateful to UK Biobank and the UK Biobank participants for making the resource data possible, and to the data processing team at Oxford University for sharing the processed data. The UK Biobank imaging project is funded by the Medical Research Council and the Wellcome Trust. Sadil, P. , & Lindquist, M. A. (2024). Comparing automated subcortical volume estimation methods; amygdala volumes estimated by FSL and FreeSurfer have poor consistency. Human Brain Mapping, 45(17), e70027. 10.1002/hbm.70027 DATA AVAILABILITY STATEMENT The data that support the findings of this study are available from UK Biobank. Restrictions apply to the availability of these data, which were used under license for this study. Data are available from https://www.ukbiobank.ac.uk/enable-your-research/apply-for-access with the permission of UK Biobank. Code to reproduce analyses is available on GitHub: https://github.com/psadil/auto-volume-comparisons. REFERENCES Associated Data Supplementary Materials Data S1. Supporting Information. Data Availability Statement The data that support the findings of this study are available from UK Biobank. Restrictions apply to the availability of these data, which were used under license for this study. Data are available from https://www.ukbiobank.ac.uk/enable-your-research/apply-for-access with the permission of UK Biobank. Code to reproduce analyses is available on GitHub: https://github.com/psadil/auto-volume-comparisons.