Replication Packages in Software Engineering Secondary Studies: A Systematic Mapping Journal: Information and Software Technology

Aleksi Huotala, Miikka Kuutila, Mika Mäntylä · 2025

Read Replication Packages in Software Engineering Secondary Studies: A Systematic Mapping Journal: Information and Software Technology by Aleksi Huotala, Miikka Kuutila, Mika Mäntylä (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12646.txt.

Introduction

Aleksi Huotala, Miikka Kuutila, and Mika Mäntylä audit how well software engineering reviews share the materials behind their conclusions. The catalog title differs from the head title, though both name the same mapping study; the head governs here. Systematic reviews summarize a field's evidence, and their replication packages, the data and scripts, decide whether anyone can check the summary. Their 2025 study examines 537 secondary studies from 2013 to 2023.

Only 31.5 percent included any research artifacts at all. The trend is improving: by 2023 about sixty-two percent shared something. Far fewer, about thirty percent, used a permanent repository with a lasting identifier, the kind of storage that survives link rot. The authors recommend that journals require artifact publication for secondary studies, arguing that a review nobody can re-run asks for trust it has not earned.

Catalog record

CREATORAleksi Huotala, Miikka Kuutila, Mika Mäntylä
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE12 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12646

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Replication Packages in Software Engineering Secondary Studies: A Systematic Mapping Journal: Information and Software Technology Aleksi Huotala, Miikka Kuutila, Mika Mäntylä Abstract Context: Systematic reviews (SRs) summarize state-of-the-art evidence in science, including software engineering (SE). Objective: Our objective is to evaluate how SRs report replication packages and to provide a comprehensive list of these packages. Method: We examined 528 secondary studies published between 2013 and 2023 to analyze the availability and reporting of replication packages. Results: Our findings indicate that only 25.4% of the reviewed studies include replication packages. Encouragingly, the situation is gradually improving, as our regression analysis shows significant increase in the availability of replication packages over time. However, in 2023, just 50.6% of secondary studies provided a replication package while an even lower percentage, 29.1% had used a permanent repository with a digital object identifier (DOI) for storage. Conclusion: To enhance transparency and reproducibility in SE research, we advocate for the mandatory publication of replication packages in secondary studies. 1 Introduction Systematic reviews (SRs) summarize state-of-the-art evidence in software engineering (SE). We argue that all SRs should include a replication package, as the true transparency of SRs is impossible without it. Transparency in systematic reviews is crucial for multiple reasons: Reason 1: Replicability (Reproducibility or Repeatability). Replication is a hallmark of science and an essential component of research integrity. Other researchers must replicate the review process to confirm its reliability and accuracy. These issues are promoted in open science [ 1 ] , open data [ 2 ] , and in the principles of Findable, Accessible, Interoperable, and Reusable (FAIR) data endorsed in Nature in 2016 [ 3 ] . Reason 2: Credibility, Trust, and Bias Reduction. Transparent SRs build trust in the integrity of the evidence. When the method, data, and process are fully available, readers can trust that the conclusions are based on a thorough and unbiased synthesis of the evidence. Reason 3: Updates. Replication packages are essential for updating SRs. Updating SRs ensures that the evidence remains current, preventing the research from becoming outdated or potentially misleading [ 4 ] . Updates are necessary when new information emerges on the topic [ 5 ] . Reason 4: Pathway to Automation. Writing an SR requires extensive time and effort – on average, 67 weeks [ 6 ] . Human errors may occur in the SR process due to reduced vigilance, but such tasks could be possibility automated. Automation can only be developed if replication packages are available to build and validate the quality of automation. Recently, large language models (LLMs) have been used to automate SRs in SE by using existing replication packages [ 7 , 8 ] . In SE research, previous studies have focused on the replication of SE experiments [ 9 ] , while reproducibility is increasingly promoted in technical areas of software engineering [ 10 ] . Based on these, we answer four research questions in relation to availability of replication packages, in Section 3 . 2 Methods We conducted a systematic mapping study following the SIGSOFT Empirical Standards checklist 1 1 1 https://www2.sigsoft.org/EmpiricalStandards/tools/ . 2.1 Search process The search was finalized on 2nd October 2024, using Scopus, which indexes metadata, including paper titles, abstracts, and keywords, from reputable scientific publishers, including all relevant SE journals. The list of journals we searched are in Figure 1 . We selected publications from 2013 to 2023, as key data repositories such as Zenodo and Figshare were launched in 2013 and 2011, respectively. The search used keywords related to systematic literature reviews, systematic mapping studies, and their corresponding variations, resulting in 628 articles. The search query we used is located in the replication package 2 2 2 https://zenodo.org/records/13918119 . 2.2 Study selection and quality assessment We applied the following inclusion criteria: 1. Study published between 2013–2023 2. Study is a secondary study 3. Study is related to SE Title–abstract screening eliminated studies that were not secondary studies or related to SE. To see if the study contains a replication package, we full-text screened each research paper. Research papers from “ACM Computing Surveys" and “Computer Science Review" journals were manually reviewed by the authors to select SE-related studies, as the two journals are not strictly limited to SE-related publications. After applying the inclusion criteria, 528 secondary studies remained. 2.3 Data extraction First, we identified dedicated sections indicating the availability of replication packages. If not found, we also conducted a text search, which consisted of various words and sentences related to external data, e.g. “Replication", “Data is available", “External" and “Zenodo", “Figshare", and “http:" (for checking any outbound links). 3 Results Figure 1: a) Comparison between publication channels and their replication packages, b) yearly statistics and c) a logistic regression model for the availability of replication packages - year is a scaled ordered factor and comparison to a reference category (journal) The results presented below are in Figure 1 . RQ1: What proportion of secondary studies include a replication package and directly link to the data? RQ1 gives an overview of the availability of replication packages in SE-related SRs and how the package is accessible. Ideally, every secondary study would include a replication package accessible through a direct link. Results: In our dataset, 134 of the 528 (25.4%) reviewed secondary studies included a replication package. 130 out of the 134 (97.1%) secondary studies that included a replication package also contained a direct link to the data. RQ2: Where are the replication packages stored? RQ2 addresses the storage location for replication packages. Replication packages should be located in permanent storage with a digital object identifier (DOI). Links to personal or institutional web pages may stop working in a few years. The use of a permanent repository (Zenodo, Figshare, or Mendeley Data) was observed to be present in less than half of the studies that included a replication package. To be precise, this was 56 out of 134 studies, amounting to 41.8% of secondary studies with a replication package while being 10.8% out of all papers (56 of 528). RQ3: How is the availability of data stated? RQ3 focuses on how the availability of replication packages is reported in research papers. Out of the studies that included a replication package, 42 of 134 (31.3%) included a dedicated section that stated the availability of data or a replication package. RQ4: How does publication year and the publication forum impact the availability of replication packages? RQ4 answers how the publication forum and year impact the availability of replication packages. Figure 1 shows a binary logistic regression model, which models whether a replication package is available, based on the publication year and journal. For the modeling, we filtered out journals that had less than 10 publications. For the reference category, which is the journal other journals are compared to, we chose “IEEE Transactions on Software Engineering", because of its esteemed reputation in the field of SE. Figure 1 shows that the publication year, which was scaled and modeled as an ordered factor for more easily interpretable effect size measure, is a significant predictor of whether a paper includes a replication package. The models shows that for every 3 years (one standard deviation), the odds of the publication having a replication package

Read the complete text