Single-Dataset Meta-Analysis For Many-Analysts And Multiverse Studies

František Bartoš, Suzanne Hoogeveen, Alexandra Sarafoglou, Samuel Pawel · 2025

Read Single-Dataset Meta-Analysis For Many-Analysts And Multiverse Studies by František Bartoš, Suzanne Hoogeveen, Alexandra Sarafoglou, Samuel Pawel (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-17064.txt.

Introduction

František Bartoš, Suzanne Hoogeveen, Alexandra Sarafoglou, and Samuel Pawel tackle a synthesis problem created by many-analysts and multiverse studies. In those studies, many teams analyze the same dataset in different reasonable ways, and results scatter. Averaging them with standard meta-analysis would double-count the single dataset and produce overconfident conclusions.

Their fix, single-dataset meta-analysis, uses a weighted likelihood that counts the dataset's information at most once. It still yields an average effect across analytic choices, plus a measure of how much analysts disagree, and it works with common software after small input changes. They test it on studies of racial bias in soccer refereeing, marital status and heart disease, and technology use and well-being. The method gives these robustness studies the careful summary statistics they lacked.

Catalog record

CREATORFrantišek Bartoš, Suzanne Hoogeveen, Alexandra Sarafoglou, Samuel Pawel
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-17064

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Single-Dataset Meta-Analysis For Many-Analysts And Multiverse Studies František Bartoš, Suzanne Hoogeveen, Alexandra Sarafoglou, Samuel Pawel Abstract Empirical claims often rely on one population, design, and analysis. Many-analysts, multiverse, and robustness studies expose how results can vary across plausible analytic choices. Synthesizing these results, however, is nontrivial as all results are computed from the same dataset. We introduce single-dataset meta-analysis, a weighted-likelihood approach that incorporates the information in the dataset at most once. It prevents overconfident inferences that would arise if a standard meta-analysis was applied to the data. Single-dataset meta-analysis yields meta-analytic point and interval estimates of the average effect across analytic approaches and of between-analyst heterogeneity, and can be supplied by classical and Bayesian hypothesis tests. Both the common-effect and random-effects versions of the model can be estimated by standard meta-analytic software with small input adjustments. We demonstrate the method via application to the many-analysts study on racial bias in soccer, the many-analysts study of marital status and cardiovascular disease, and the multiverse study on technology use and well-being. The results show how single-dataset meta-analysis complements the qualitative evaluation of many-analysts and multiverse studies. Keywords Evidence synthesis, robustness, dependence, single dataset, crowdsourced analysis, multi-analyst 1 Introduction Studies are often conducted using a single specific population, manipulation, and statistical analysis. While researchers may aim to draw broader conclusions from such work, these generalizations may not be justified unless the potential variability across populations, design choices, and analytical choices is explicitly accounted for in the analysis ( 71 ; 29 ) . A single study is usually insufficient to capture all sources of variability. Generalizations with respect to populations and designs can be assessed through cross-cultural studies and conceptual replications ( 46 ; 2 ) , while robustness across similar populations and nearly identical designs is evaluated through direct replications, for instance, via large-scale multi-site replications (e.g., Many labs studies; 37 ; 38 ; 16 ; 39 ; 17 ) and initiatives to systematically reproduce and replicate research findings (e.g., 12 ). These efforts involve collecting data from independent samples and are typically analyzed using standard meta-analytic methods. Similarly, in medicine, bottom-up collaborative approaches have gained popularity, where multiple randomized controlled trials are independently designed but follow closely aligned protocols and are jointly analyzed in a prospective meta-analysis ( 57 ; 64 ) . Generalization with respect to statistical analyses can be assessed through many-analysts studies, multiverse studies, or robustness analyses. In many-analysts studies, different analyses performed by independent researchers on a single dataset allow researchers to assess the robustness of the findings across plausible analytic strategies ( 59 ) . In multiverse studies, key aspects of the analytic strategy, such as the operationalization of outcome variables or exclusion criteria, are systematically varied to create a set of plausible analytic combinations, all of which are executed and transparently reported ( 62 ; 60 ) . On a smaller scale, robustness analyses explore the variability of results with respect to a select few choices, for instance, different shapes of prior distributions within the Bayesian analysis framework ( 69 ) . Both many-analysts and multiverse studies help reveal the “garden of forking paths”, where numerous analytic strategies may be explored but only those leading to significant results are reported ( 18 ) . In addition, the many-analysts approach allows researchers to incorporate diverse perspectives and add nuance to findings by comparing results across different expert teams or theoretical frameworks, even with a small number of analysts ( 31 ; 67 ; 5 ) . Although many-analysts and multiverse approaches have gained popularity in recent years (e.g., 30 ; 28 ; 63 ; 70 ; 32 ), statistical methods for aggregating evidence in these projects remain underdeveloped. Some recent techniques allow for hypothesis testing within the multiverse framework ( 43 ; 20 ; 14 ; 60 ) . However, these techniques require costly simulation approaches and are typically only feasible when all analyses have been developed by a single research team (as in multiverse or robustness analyses). They become prohibitively demanding in many-analysts studies. In these cases, the lead team would need to re-implement and rerun the models from all analysis teams, an effort that is often infeasible due to the substantial variation in complexity of the submitted analytic approaches. Moreover, beyond these practical limitations, these techniques cannot synthesize effect size estimates into an overall effect across analysis pipelines, nor can they quantify evidence for between-analyst heterogeneity. Synthesis of multiple effect sizes into an overall effect size is usually within the domain of meta-analytic methodology. The standard meta-analysis however assumes that effect sizes are independent and that they come from separate, non-overlapping datasets. In many-analysts studies, by contrast, all effect size estimates are derived from the same dataset, which introduces dependence among estimates and shared sources of variance. Consequently, synthesizing evidence across analytic approaches remains a major open challenge ( 1 ) . In the absence of such methods, visual representations are often recommended as an effective tool to present findings. And indeed, the majority of many-analysts studies rely on visualizations–such as forest plots–to summarize the evidence, and researchers often draw conclusions about the presence or absence of an effect based on descriptive statistics alone. This approach, though widely used, is ultimately unsatisfactory. Relying exclusively on descriptive approaches makes it impossible to formally test hypotheses of interest, for instance, whether there is a genuine effect of an experimental manipulation. Nevertheless, some projects, such as 23 and 15 , have applied standard meta-analysis to calculate an overall effect size across analysis teams, despite the methodological limitations posed by using a single underlying dataset. When standard meta-analysis is applied under these conditions, it may still yield a valid estimate of between-analyst heterogeneity; however, the uncertainty around the overall effect size will likely be underestimated, as the method incorrectly assumes that each analysis team contributes new, independent data. 1.1 The Current Manuscript In this manuscript, we introduce a meta-analytic approach for addressing the shared use of data in many-analysts studies, multiverse analyses, and, more broadly, robustness analyses. This framework enables researchers to synthesize evidence across analysis strategies in a statistically coherent manner. It is particularly valuable when the aim is to quantify analytic variability while also drawing overall conclusions about the magnitude and presence of the effect of interest. In doing so, it supports the use of many-analysts and multiverse designs to address substantive research questions and formally test hypotheses. The approach also contributes to a more comprehensive understanding of the evidence these projects generate and complements other tools for extracting meaningful insight, such as the Subjective Evidence Evaluation Survey ( 55 ) , which captures analysis teams’ beliefs about an effect’s plausibility, confidence in their results, perceived robustness, and evaluations of the dataset and study design. In the following sections, we first describe the limitations of applying standard meta-analytic techniques to many-analysts and multiverse projects—specifically, the problem of multiple estimates derived from the same dataset resulting in overly confident conclusions. Next, we outline several desirable properties for a method that quantitatively synthesizes results when effect sizes are computed from shared data. We then introduce single-dataset meta-analysis, a two-stage procedure that meets these criteria: first estimating heterogeneity across analysis teams or analytic strategies, and then pooling the

Read the complete text