Living Synthetic Benchmarks: A Neutral and Cumulative Framework for Simulation Studies

František Bartoš, Samuel Pawel, Björn S. Siepe · 2025

Read Living Synthetic Benchmarks: A Neutral and Cumulative Framework for Simulation Studies by František Bartoš, Samuel Pawel, Björn S. Siepe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19489.txt.

Introduction

František Bartoš, Samuel Pawel, and Björn S. Siepe propose living synthetic benchmarks as a fairer way to judge statistical methods. Simulation studies usually test methods on data-generating setups invented by the methods' own authors, which rewards favorable setups. Results also fragment across mismatched setups, so studies are hard to compare.

Under their proposal, benchmarks stay open and keep growing: whenever a new data-generating mechanism, method, or performance measure appears, it joins the benchmark. Method development and benchmark design are kept apart to protect neutrality. The authors lay out a blueprint for building and running such benchmarks and show feasibility with a working prototype for publication bias adjustment methods.

Catalog record

CREATORFrantišek Bartoš, Samuel Pawel, Björn S. Siepe
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-19489

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Living Synthetic Benchmarks: A Neutral and Cumulative Framework for Simulation Studies František Bartoš, Samuel Pawel, Björn S. Siepe Abstract Simulation studies are widely used to evaluate statistical methods. However, new methods are often introduced and evaluated using data-generating mechanisms (DGMs) devised by the same authors. This coupling creates misaligned incentives, e.g., the need to demonstrate the superiority of new methods, potentially compromising the neutrality of simulation studies. Furthermore, results of simulation studies are often difficult to compare due to differences in DGMs, competing methods, and performance measures. This fragmentation can lead to conflicting conclusions, hinder methodological progress, and delay the adoption of effective methods. To address these challenges, we introduce the concept of living synthetic benchmarks . The key idea is to disentangle method and simulation study development and continuously update the benchmark whenever a new DGM, method, or performance measure becomes available. This separation benefits the neutrality of method evaluation, emphasizes the development of both methods and DGMs, and enables systematic comparisons. In this paper, we outline a blueprint for building and maintaining such benchmarks, discuss the technical and organizational challenges of implementation, and demonstrate feasibility with a prototype benchmark for publication bias adjustment methods. We conclude that living synthetic benchmarks have the potential to foster neutral, reproducible, and cumulative evaluation of methods, benefiting both method developers and users. Keywords : Evidence synthesis, method benchmarking, method comparison, systematic review 1 Introduction “ In fact it is very difficult to run an honest simulation comparison, and easy to inadvertently cheat by choosing favorable examples, or by not putting as much effort into optimizing the dull old standard as the exciting new challenger. ” Bradley Efron commenting on Breiman 2001 Simulation studies play a crucial role in methodological research, allowing researchers to systematically test and compare statistical (data analysis) methods under controlled conditions ( Hoaglin & Andrews 1975 ) . In such studies, researchers simulate synthetic datasets based on a known data-generating mechanism (DGM), analyze these data using the methods under investigation, and compare their outputs with the known truth to evaluate method performance (see Figure 1 for an illustration). Ground Truth Simulated Data Analysis Output Data-Generating Mechanism Data Analysis Performance Evaluation Methods Linear regression Random forest \ldots Parameters Sample size Variance \ldots Measures Bias Mean squared error \ldots Figure 1: Schematic illustration of a simulation study. In contrast to purely theoretical assessments, simulation studies can reveal practical strengths, limitations, and boundary conditions of statistical methods under complex conditions. However, as in empirical research in other scientific fields, conflicts of interest and misaligned incentives can compromise the validity of simulation studies. Researchers developing new statistical methods are often expected to demonstrate the new method’s superior performance in a simulation study, resulting in pressure to demonstrate the new method in the best light ( Norel et al. 2011 ; Boulesteix et al. 2015 ; Boulesteix et al. 2020b ; Mandl et al. 2025 ) . This pressure is particularly concerning given that simulation studies in methodological research have been shown to suffer from design, execution, and reporting issues almost since their inception ( Hoaglin & Andrews 1975 ; Hauck & Anderson 1984 ; Koehler et al. 2009 ; Harwell et al. 2018 ; Morris et al. 2019 ; Siepe et al. 2024 ; Williams et al. 2024 ) . In the design of a simulation study, researchers may often rely on ad-hoc or oversimplified parametric DGMs to simulate data ( Boulesteix et al. 2020a ; Sauer et al. 2025 ) . The simulation study introducing a new method may be prone to using DGMs that favor the novel method’s performance ( Ullmann et al. 2022 ; Nießl et al. 2022 ; Breiman 2001 ; Nießl et al. 2024 ) . Small changes to the DGM can produce markedly different outcomes, complicating method evaluations and reducing generalizability ( Pateras et al. 2018 ; van Smeden et al. 2016 ; Fairchild et al. 2024 ) . Ambiguity in the objectives and evaluation criteria of a simulation study leaves room for flexible execution and reporting choices that might bias the between-method comparison. There are many researchers’ degrees of freedom in the execution of a simulation study ( Pawel et al. 2024 ) . When different methods are compared, researchers may be less familiar with methods that they did not develop themselves ( Nießl et al. 2024 ) . This can lead to a suboptimal implementation of competitor methods, again favoring the method of choice. Due to computational issues or non-convergence of a method, simulation repetitions may not always lead to valid outputs ( Pawel et al. 2025 ; Wünsch et al. 2025 ) . Different ways of handling such issues can change the main results and implications of a simulation study. Many other small decisions, such as the choice of hyperparameters, the removal of certain conditions, or a change in the evaluation criteria, can also hinder an impartial evaluation ( Pawel et al. 2024 ) . Reporting issues in methodological research are similar to those found in other empirical research ( Hoaglin & Andrews 1975 ; Hauck & Anderson 1984 ; Harwell et al. 2018 ) . Simulation studies often contain a large number of results, and selective reporting of outcomes can easily favor a specific method ( Pawel et al. 2024 ) . Rather than weighing the trade-offs between different methods in different contexts, simulation results are sometimes reported in a way that seeks to find the “best” overall method, which is often unrealistic and unhelpful ( Strobl & Leisch 2024 ) . The Monte Carlo uncertainty of performance estimates is rarely reported in simulation studies, which is especially problematic in computationally intensive studies using relatively few simulation repetitions ( Morris et al. 2019 ; Siepe et al. 2024 ; Koehler et al. 2009 ) . Several literature reviews have found that code to reproduce a simulation is often not shared and that insufficient information on the computational environment is provided ( Siepe et al. 2024 ; Morris et al. 2019 ; Williams et al. 2024 ; Kucharský et al. 2020 ) . This hinders the reproducibility and replicability of simulation studies, and the reuse of design choices in new studies ( Luijken et al. 2024 ; Lohmann et al. 2022 ) . Overall, these issues lead to a proliferation of simulation studies that introduce, evaluate, and favor novel methods. Consequently, different contributions to statistical methodology result in conflicting conclusions and recommendations ( Fairchild et al. 2024 ; Lang & Bliese 2024 ) . One might speculate that those discrepancies confuse empirical researchers who have to choose a method for analyzing their data, result in the delayed adoption of effective statistical approaches in practice, and hinder further methodological advancements ( Lotterhos et al. 2022 ; Boulesteix et al. 2025 ) . To overcome these issues, we propose to disentangle the development of statistical methods and the design of simulation studies via the adoption of living synthetic benchmarks . This proposal is inspired by recent advances in machine learning and artificial intelligence research; standardized benchmarks such as the ImageNet ( Deng et al. 2009 ) and Critical Assessment of Structure Prediction ( Moult et al. 1995 , CASP, ) competitions have resulted in groundbreaking scientific

Read the complete text