Towards an OSF-based Registered Report Template for Software Engineering Controlled Experiments Conference: 3rd International Workshop on Methodological Issues with Empirical Studies in Software Engineering ; April 12–18, 2026; Rio de Janeiro, Brazil 3rd International Workshop on Methodological Issu
Ana B. M. Bett, Thais S. Nepomuceno, Edson OliveiraJr, Maria Teresa Baldassare, Valdemar V. Graciano Neto, Marcos Kalinowski · 2026
Read Towards an OSF-based Registered Report Template for Software Engineering Controlled Experiments Conference: 3rd International Workshop on Methodological Issues with Empirical Studies in Software Engineering ; April 12–18, 2026; Rio de Janeiro, Brazil 3rd International Workshop on Methodological Issu by Ana B. M. Bett, Thais S. Nepomuceno, Edson OliveiraJr, Maria Teresa Baldassare, Valdemar V. Graciano Neto, Marcos Kalinowski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09292.txt.
Introduction
Ana B. M. Bett and five co-authors work toward a Pre-filed Report form for set trials in software building, hosted on the Open Science Framework. Pre-filed Reports pre-file guesses, ways, and reviews for peer review before data gathering, curbing p-hacking, print slant, and unfit after-the-fact review. Yet trial write-ups in the field still lack bite, hurting rerun-ability and openness.
The team read chosen OSF Pre-filed Report types against write-up rules for set trials. One type matched many rule hints, but none covered them fully, and OSF form tweaking proved tight. The paper ends that taking up OSF-based Pre-filed Reports pays off, but setting report-specific rules for software-building trials remains needed work.
Catalog record
| CREATOR | Ana B. M. Bett, Thais S. Nepomuceno, Edson OliveiraJr, Maria Teresa Baldassare, Valdemar V. Graciano Neto, Marcos Kalinowski |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-09292
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards an OSF-based Registered Report Template for Software Engineering Controlled Experiments Conference: 3rd International Workshop on Methodological Issues with Empirical Studies in Software Engineering ; April 12–18, 2026; Rio de Janeiro, Brazil 3rd International Workshop on Methodological Issues with Empirical Studies in Software Engineering (WSESE ’26), April 12–18, 2026, Rio de Janeiro, Brazil DOI: 10.1145/3786149.3788308 ISBN: 979-8-4007-2382-7/2026/04 Price: 15.00 DOI: 10.1145/3530019.3535306 ISBN: 978-1-4503-9613-4/22/06 Ana B. M. Bett, Thais S. Nepomuceno, Edson OliveiraJr, Maria Teresa Baldassare, Valdemar V. Graciano Neto, Marcos Kalinowski Abstract . Context: The empirical software engineering (ESE) community has contributed to improving experimentation over the years. However, there is still a lack of rigor in describing controlled experiments, hindering reproducibility and transparency. Registered Reports (RR) have been discussed in the ESE community to address these issues. A RR registers a study’s hypotheses, methods, and/or analyses before execution, involving peer review and potential acceptance before data collection. This helps mitigate problematic practices such as p-hacking, publication bias, and inappropriate post hoc analysis. Objective: This paper presents initial results toward establishing an RR template for Software Engineering controlled experiments using the Open Science Framework (OSF). Method: We analyzed templates of selected OSF RR type in light of documentation guidelines for controlled experiments. Results: The observed lack of rigor motivated our investigation of OSF-based RR types. Our analysis showed that, although one of the RR types aligned with many of the documentation suggestions contained in the guidelines, none of them covered the guidelines comprehensively. The study also highlights limitations in OSF RR template customization. Conclusion: Despite progress in ESE, planning and documenting experiments still lack rigor, compromising reproducibility. Adopting OSF-based RRs is proposed. However, no currently available RR type fully satisfies the guidelines. Establishing RR-specific guidelines for SE is deemed essential. † † cc-license: by 1. Introduction Software engineering (SE) research still faces numerous challenges, one of which consists of obtaining and managing research artifacts ( 9 ) . Rigorous research is mainly based on transparency, credibility, and reproducibility, which are pillars in constructing and evaluating robust theories for the area’s evolution ( 10 ) . Open Science principles and practices can provide a solid foundation for achieving these goals 1 1 1 https://unesdoc.unesco.org/ark:/48223/pf0000379949 ( 7 ) . In recent years, several major SE conferences, such as ICSE 2 2 2 https://conf.researchr.org/track/icse-2025/icse-2025-artifact-evaluation and FSE 3 3 3 https://conf.researchr.org/track/fse-2025/fse-2025-artifacts , have introduced Artifact Tracks. These tracks focus on evaluating and promoting the quality and availability of research artifacts, such as software tools, data, and frameworks that support the reproducibility and validity of research findings. The conferences share common goals and inclusion criteria: artifacts must be functional, reusable, and publicly available in persistent repositories with proper documentation. By incorporating these tracks, the conferences highlight the importance of artifacts in ensuring the transparency, reliability, and reproducibility of empirical studies in software engineering. To emphasize artifact quality, many conferences, including ICSE and FSE, collaborate with the ACM’s badging system. This system awards badges to artifacts that meet specific criteria, such as functionality and reusability. These badges recognize high-quality artifacts that particularly enhance the reproducibility and reusability of research. Additionally, the ACM badges encourage the community to prioritize transparency and the open sharing of research results, ensuring that artifacts are accessible and valuable for researchers seeking to replicate or build upon existing findings. Although reproducibility and reliability are crucial, many Software Engineering studies still face challenges, including issues with replicability, biases in results, and incomplete research artifacts. In response to these challenges, Registered Reports (RRs) have emerged as a promising solution, as they enhance research quality by requiring hypotheses and study protocols to be established prior to data collection, thereby reducing bias and increasing transparency. RRs are peer-reviewed, which straightforwardly improves such characteristics. RRs are being adopted at several SE conferences, including ICSE, ESEM, ICSME, and MSR. Recent studies have addressed this topic, including that of 4 . However, this topic remains crucial for the reproducibility, reliability, and transparency of empirical studies. In this paper, we focus on the following research question: “ how can we design an RR template for SE controlled experiments based on existing guidelines? ”. To do so, we propose and discuss an initial template based on the Open Science Framework (OSF) platform. OSF offers various RR templates for different kinds of empirical studies. Therefore, we analyzed and compared all of them to determine how well they align with the specific guidelines for SE-controlled experiments compiled by 6 . We also discuss the limitations of this study, share the lessons learned, and suggest potential future actions. This paper is organized as follows: Section 2 presents the analyzed controlled experimentation guidelines; Section 3 presents essential background on Registered Reports and the Open Science Framework platform used for proposing an RR template; Section 4 presents our methodology, the proposed initial RR template, and discusses lessons learned and limitations; Section 5 presents prospective actions based on the proposed template; and Section 6 presents final remarks. 2. Guidelines for Documenting Controlled Experiments and RR in SE 6 present a set of guidelines for experiments in SE. These guidelines provide detailed guidance on the expected content of sections and subsections for reporting controlled experiments and quasi -experiments. The guidelines are organized into sections, each one focusing on a core component of controlled experiment reporting in SE. These sections cover structured abstract, introduction, experiment planning, execution, analysis, interpretation, and conclusions. Table 1 summarizes these guidelines by section and scope. Table 1 . Experiment Guidelines by 6 ID Section Sub-Section Scope G1 Title <title>+ "- A controlled experiment" G2 Authorship Does it include contact information? G3 Structured Abstract Background Why is this research important? G4 Structured Abstract Objective What is the question addressed with this research? G5 Structured Abstract Methods What is the statistical context and methods applied? G6 Structured Abstract Results What are the main findings? Practical implications? G7 Structured Abstract Limitations What are the weaknesses of this research? G8 Structured Abstract Conclusions What is the conclusion? G9 Keywords Areas of research, treatments, dependent variables, and study type G10 Introduction Problem Statement What is the problem? Where does it occur? Who has observed it? Why is it important? G11 Introduction Research Objectives Analyze <Object(s) of study>for the purpose of <purpose> G12 Introduction Context What are the environmental factors that impact generalizability? G13 Related Work How does it relate to existing research or state of the practice? G14 Experiment Planning Goals Formalization of goals, defining important constructs G15 Experiment Planning Experimental Units From which population will the sample be drawn? G16 Experiment Planning Experimental Material Which objects are selected and why? G17 Experiment Planning Tasks Which tasks have to be performed by the subjects? G18 Experiment Planning Hypotheses, Variables Formalization of hypotheses and variables G19 Experiment Planning Design What type of experimental design was chosen? G20 Experiment Planning Procedure How will the experiment (data collection) be performed? G21 Experiment Planning Analysis Procedure How will the data be analyzed? G22 Execution Main purpose: describe any deviations from the plan G23 Execution Preparation What was done to prepare for the execution of the experiment? G24 Execution Deviations Describe any deviations from the plan, e.g., data collection G25 Analysis Descriptive Statistics What are the results from descriptive statistics? G26 Analysis Data Set