On the reproducibility of discrete-event simulation studies in health research: an empirical study using open models
Amy Heather a , Thomas Monks a , Alison Harper b , Navonil Mustafee b , Andrew Mayne c · 2025
Read On the reproducibility of discrete-event simulation studies in health research: an empirical study using open models by Amy Heather a , Thomas Monks a , Alison Harper b , Navonil Mustafee b , Andrew Mayne c (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-13137.txt.
Introduction
Amy Heather and co-authors test whether published health simulations can be rerun in a 2025 paper. Discrete-event simulation models guide health care choices, yet their code is rarely checked. The team picks eight open models in Python or R across varied topics and years. Each model is rebuilt from its public files and paper.
Half the models fully reproduce, they report, and half only partly, down to about an eighth of stated outcomes. Fixes take up to twenty-eight hours per model. Missing licences, code that differs from the paper, and absent output scripts are the main blocks. Simple author habits would cure most of it, they argue.
Catalog record
| CREATOR | Amy Heather a , Thomas Monks a , Alison Harper b , Navonil Mustafee b , Andrew Mayne c |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-13137
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the reproducibility of discrete-event simulation studies in health research: an empirical study using open models Amy Heather a , Thomas Monks a , Alison Harper b , Navonil Mustafee b , Andrew Mayne c Abstract Reproducibility of computational research is critical for ensuring transparency, reliability and reusability. Challenges with computational reproducibility have been documented in several fields, but healthcare discrete-event simulation (DES) models have not been thoroughly examined in this context. This study assessed the computational reproducibility of eight published healthcare DES models (Python or R), selected to represent diverse contexts, complexities, and years of publication. Repositories and articles were also assessed against guidelines and reporting standards, offering insights into their relationship with reproducibility success. Reproducing results required up to 28 hours of troubleshooting per model, with 50% fully reproduced and 50% partially reproduced (12.5% to 94.1% of reported outcomes). Key barriers included the absence of open licences, discrepancies between reported and coded parameters, and missing code to produce model outputs, run scenarios, and generate tables and figures. Addressing these issues would often require relatively little effort from authors: adding an open licence and sharing all materials used to produce the article. Actionable recommendations are proposed to enhance reproducibility practices for simulation modellers and reviewers. keywords Discrete-event simulation; health; reproducibility; open science; open models † † affiliation: a University of Exeter Medical School, St Luke’s Campus, Heavitree Rd, Exeter, UK; b Centre for Simulation, Analytics and Modelling (CSAM), University of Exeter Business School, Streatham Campus, Rennes Drive, Exeter, UK; c Somerset NHS Foundation Trust, Taunton, UK Abstract word count: 178 Article word count: 8013 1 Introduction The reproducibility and replicability of published research are widely recognised as cornerstones of rigorous science and have been investigated across numerous disciplines. 1 Within the field of modelling and simulation, the importance of reproducibility has gained increasing attention and is recognised as essential for ensuring the reliability and impact of simulation-based research. 2 , 3 , 4 Despite this growing emphasis, no study has empirically assessed the reproducibility of results generated by healthcare discrete-event simulation (DES) models. These models are important tools in healthcare decision-making, aiming to enhance patient outcomes and optimise health service delivery by addressing challenges such as patient flow management, resource allocation, and cost-effectiveness analyses. 5 This study investigates the reproducibility of open DES models - those with code made publicly available for others to run and reuse - with an emphasis on identifying barriers and facilitators to reproduction. The study identifies the work required to reuse the author-supplied model code to reproduce the simulation results presented in the tables, charts, and text of journal articles, including any troubleshooting required during the process. The study aims to provide actionable guidance to simulation modellers to improve the reproducibility of their work. 1.1 Why reproducibility matters Reproducibility is the ability to regenerate published results using the provided code and data. Failures in reproducibility may indicate underlying issues, such as incorrect parameters, missing or outdated code, or unexpected changes in software behaviour due to updates. Achieving reproducibility fosters trust and transparency, building confidence that the model behaves consistently with the original implementation. It is vital to establish reproducibility before a model is reused. 6 , 7 , 8 Reuse is the adaptation and application of a model in new contexts, amplifying the potential real-world impact of the model. Reuse is in high demand; for example, 37% of Wellcome Trust-funded researchers report using external code, often from colleagues or community repositories. 9 As shown in Figure 1 , reproducibility exists within a continuum of code attributes, including being re-runnable, repeatable, reusable and replicable. Together, these characteristics underpin the validity and reliability of simulation models. 10 Reproducibility also benefits authors by making it easier to revisit and reuse their own code, such as for updating outputs or conducting new analyses. 7 Troubleshooting non-reproducible code can be time-consuming or even impossible if required information is now lost. 6 , 7 For instance, code may need to be re-run following peer review. 7 Given that the mean time from submission to publication in biomedical research ranges from 27 to 639 days, 11 it is important that code remains functional and reproducible long after its initial execution. Even if authors retain access to their code, failure to record exact parameters for every scenario or document the computational environment used, for example, could make the code non-reproducible when revisited, even if only a few months later. Finally, reproducibility often improves overall code quality. It encourages clear structure and documentation, reducing errors and ambiguities. 7 These advantages apply regardless of whether code is shared publicly or remains proprietary. Figure 1: Five standards for scientific code, as described in Benureau et al. (2018). 10 1.2 Addressing challenges in reproducibility The computational reproducibility of published research has been investigated across numerous disciplines. These studies, which focus on the reproducibility of results using shared code and data, vary in scope and methodology - for example, the extent of troubleshooting allowed when running code. Across many of these studies though, a consistent finding is that the majority of attempts to reproduce results fail, with examples in epidemiology, 12 public health, 13 economics, 14 , 15 political science, 16 physics, 17 , 18 and hydrology. 19 However, some studies do observe higher rates of reproducibility, often under specific conditions. Investigations in political science, 20 psychology, 21 and articles published in Science 22 found that while many papers lack accessible data or code, the majority of results could be reproduced in those that provide these resources, in these cases. Similarly, studies in geoscience 23 and a large-scale analysis of articles with Jupyter notebooks 24 found that artefacts may often fail to execute, but when they do, the majority of results could be reproduced. High reproducibility was observed in management science, 3 where 95% of studies were fully or largely reproduced - though this dropped to 68% when including studies with inaccessible datasets. Journals can play a pivotal role in addressing reproducibility issues, as summarised in Table 1.2 . Simple initiatives include requiring code availability statements or requiring that code is shared upon request. However, authors may simply state that code is unavailable or choose not to share it. 22 , 25 For example, in a study of individual- and agent-based simulation models, researchers emailed authors who had stated in journal articles that their code was available upon request. However, they received responses from less than 1% of authors - some provided links to the code, but others stated they no longer had access to it. 25 To address this, journals may mandate code sharing via public repositories or archives. 26 , 27 , 28 , 29 , 30 , 3 , 31 , 32 However, code sharing alone does not guarantee reproducibility, so journals may also review repositories to check that code is documented, complete and/or likely to be executable. 33 , 32 , 31 , 26 , 28 Some journals attempt to run the provided code during peer review to assess its computational reproducibility, 32 , 34 , 26 , 28 or accept reproductions of published work by other groups as formal publications. 35 When code or data cannot be shared for ethical or legal