Code Sharing In Prediction Model Research: A Scoping Review

Thomas Sounack, Raffaele Giancotti, Catherine A. Gao, Lasai Barreñada, Hyeonhoon Lee, Hyung-Chul Lee, Leo Anthony Celi, Karel G.M. Moons, Gary S. Collins, Charlotta Lindvall, Tom Pollard · 2026

Read Code Sharing In Prediction Model Research: A Scoping Review by Thomas Sounack, Raffaele Giancotti, Catherine A. Gao, Lasai Barreñada, Hyeonhoon Lee, Hyung-Chul Lee, Leo Anthony Celi, Karel G.M. Moons, Gary S. Collins, Charlotta Lindvall, Tom Pollard (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06212.txt.

Introduction

Sounack and ten colleagues review code sharing in diagnostic and prognostic prediction model research, where code is essential for reproduction yet rarely available. Screening PubMed articles citing TRIPOD or TRIPOD+AI, they assess 3,967 eligible studies with a language-model pipeline checking repo links against 14 features. Of those, 12.2 percent shared code, rising to about 16 percent in 2025. Shared repos usually hold a README but often lack dependency lists or modular structure, setting a baseline for the planned TRIPOD-Code guideline.

Catalog record

CREATORThomas Sounack, Raffaele Giancotti, Catherine A. Gao, Lasai Barreñada, Hyeonhoon Lee, Hyung-Chul Lee, Leo Anthony Celi, Karel G.M. Moons, Gary S. Collins, Charlotta Lindvall, Tom Pollard
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49814
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.06212

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Code Sharing In Prediction Model Research: A Scoping Review Thomas Sounack, Raffaele Giancotti, Catherine A. Gao, Lasai Barreñada, Hyeonhoon Lee, Hyung-Chul Lee, Leo Anthony Celi, Karel G.M. Moons, Gary S. Collins, Charlotta Lindvall, Tom Pollard Abstract Background Analytical code is essential for reproducing diagnostic and prognostic prediction model research, yet code availability in the published literature remains limited. While the TRIPOD statements set standards for reporting prediction model methods, they do not define explicit standards for repository structure and documentation. This review quantifies current code-sharing practices to inform the development of TRIPOD-Code, a TRIPOD extension reporting guideline focused on code sharing. Methods We conducted a scoping review of PubMed-indexed articles citing TRIPOD or TRIPOD+AI as of Aug 11, 2025, restricted to studies retrievable via the PubMed Central Open Access API. Eligible studies developed, updated, or validated multivariable prediction models. A large language model (LLM)-assisted pipeline (GPT-5.2 2025-12-11) was developed to screen articles and extract code availability statements and repository links. Repositories were downloaded using a code retrieval utility and assessed with the same LLM against 14 predefined reproducibility-related features, including the presence of a README file, versioned dependencies, and modularity. Our code is made publicly available 1 . Findings Among 3,967 eligible articles, 12.2% (n=482) included code sharing statements. Code sharing increased over time, reaching 15.8% in 2025, and was higher among TRIPOD+AI-citing studies than TRIPOD-citing studies (29.7% vs. 11.8% in 2025, excluding articles citing both). Sharing prevalence varied widely by journal and country, with higher sharing observed in journals that discuss code availability in their submission guidelines. Of 447 extracted repository links, 83.4% were hosted on GitHub. Repository assessment showed substantial heterogeneity in reproducibility features: most repositories contained a README file (80.5%), but fewer specified dependencies (37.6%; version-constrained 21.6%) or were modular (42.4%). Data or sample datasets were included in 37.1% of repositories, and among the 286 repositories that included stochastic components, 64.3% fixed random seeds. Interpretation In prediction model research, code sharing remains relatively uncommon, and when shared, often falls short of being reusable. These findings provide an empirical baseline for the TRIPOD-Code extension and underscore the need for clearer expectations beyond code availability, including documentation, dependency specification, licensing, and executable structure. † † 1 https://github.com/thomas-sounack/TRIPOD-Code 1 Introduction Development and evaluation of diagnostic and prognostic models are inherently computational. The analytical code used to, e.g., preprocess data, develop the prediction modeling approach, select or tune model parameters, and evaluate performance is essential for methodological assessment and reproduction of results. Yet access to the code underlying published prediction models remains uncommon [ 8 ] . Concerns about computational transparency in scientific research are longstanding, and recent initiatives have sought to strengthen reporting and sharing practices. The FAIR principles have outlined standards for reusable digital research artifacts [ 32 ] ; the EQUATOR Network has advanced reporting guidelines to enhance methodological transparency [ 25 ] ; and journals and funders increasingly require data and code availability statements to promote reproducibility [ 10 ; 2 ; 5 ] . Within clinical prediction research, the TRIPOD statement and its update in 2024, TRIPOD+AI (www.tripod-statement.org) [ 7 ; 16 ; 6 ] , were developed to improve the completeness and clarity of model reporting. These guidelines have helped promote methodological quality, external validation, and calibration of prediction model research [ 34 ] . However, although TRIPOD+AI encourages sharing of analytical code, it does not explicitly specify standards for repository structure, documentation, or reproducibility. Consequently, reporting of code availability is likely to be heterogeneous [ 28 ] , although the actual extent and quality of code sharing in prediction model research have not yet been studied. Evidence from broader biomedical literature indicates that code sharing is infrequent and that declared code availability often does not correspond to functional accessibility [ 12 ; 24 ] . In a large-scale attempt to rerun Jupyter notebooks linked to biomedical articles in PubMed Central, prior work identified thousands of Python Jupyter notebooks but found that only 5.3% could be run successfully end-to-end [ 23 ] . Manual review of code repositories reports similar gaps: among articles accepted at the Medical Imaging with Deep Learning (MIDL) conference, only 22% had a public repository judged “repeatable” under predefined reproducibility criteria [ 26 ] , and in a study evaluating 160 deep learning computational pathology articles, only about one quarter were found to make code publicly available [ 31 ] . However, these investigations focused on specific code formats (e.g., Jupyter notebooks) or assessed code documentation through a labor-intensive manual review that does not scale to large, diverse corpora. Prior work has shown the potential of large language models (LLMs) to facilitate large-scale reviews [ 3 ; 9 ] , but to our knowledge, no study has assessed code sharing and repository documentation practices in prediction model research, applied LLMs to jointly extract code-sharing statements from articles, or characterized repository features at scale. Establishing this empirical baseline is necessary to understand the current practice of code sharing in prediction model research and to inform the development of more precise reporting standards. In response to this need, we are developing the TRIPOD-Code extension to provide structured guidance on code availability and computational reproducibility in prediction model research [ 21 ] . The present study represents Stage 1 of this project: a large-scale scoping review of studies citing TRIPOD or TRIPOD+AI, undertaken to quantify code availability and evaluate the characteristics of shared repositories. 2 Methods We conducted a scoping review to characterize code sharing practices for papers developing or evaluating multivariable prediction models in healthcare. The review addresses two key questions: (1) What proportion of multivariable prediction model studies that cite the TRIPOD or TRIPOD+AI statements report on the availability of analytical code? (2) Among studies providing accessible code, what are the structural and documentation characteristics of the code? The review was conducted in accordance with the PRISMA extension for scoping reviews [ 30 ] ( Appendix A ). The protocol for this review was registered (INPLASY202620080) [ 27 ] . 2.1 Search strategy and selection criteria The cohort comprised all PubMed-indexed articles that cited the TRIPOD or TRIPOD+AI statements (www.tripod-statement.org) as of August 11th, 2025. Focusing on TRIPOD-citing studies provides a defined and policy-relevant subset of prediction model research in which reporting guidance is already acknowledged. As such, this sampling frame represents a conservative estimate of code sharing practices among studies attentive to reporting standards. Due to simultaneous publication across different journals, TRIPOD and TRIPOD+AI have multiple entries in PubMed (12 and 2 entries, respectively). For each entry, citation lists were downloaded from the PubMed “Cited by” section and aggregated programmatically. Duplicate records were removed. We included primary research articles that developed, validated, or updated a multivariable prediction model using statistical or machine learning methods, consistent with definitions in the TRIPOD-Code protocol [ 21 ] . To enable reproducibility without subscription barriers, we restricted inclusion to articles retrievable through the PubMed Central Open Access API [ 20 ] . No additional exclusion criteria were applied at this stage. 2.2 Data analysis Data analysis comprised two components: (1) article-level screening and metadata extraction and (2) characterization of associated code repositories. An LLM-assisted pipeline was implemented using a predefined structured output schema (Appendices B

Read the complete text