An Agentic Approach Towards Replication Package Quality Evaluation
Maximilian Alexander Amougou Mbida, Florian Angermeir · 2026
Read An Agentic Approach Towards Replication Package Quality Evaluation by Maximilian Alexander Amougou Mbida, Florian Angermeir (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02006.txt.
Introduction
Amougou Mbida and Angermeir present an early study of software research artifacts. Their focus is replication packages. These packages should let others repeat a study and reuse its materials. Yet checks of their quality are still mostly done by hand. Such checks are hard to scale up.
They turn open science rules into clear tests that a computer can check. From 380 rules in 34 sources, they build 51 criteria. They make 31 of them work for automated checks. A team of agents then reads a package and writes a report backed by evidence. In tests on five packages, the authors report high consistency across runs. They also report weaker results on studies that use qualitative methods. A small survey suggests researchers saw the tool as useful.
Catalog record
| CREATOR | Maximilian Alexander Amougou Mbida, Florian Angermeir |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.02006
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Agentic Approach Towards Replication Package Quality Evaluation Maximilian Alexander Amougou Mbida, Florian Angermeir Abstract Reproducibility in empirical software engineering relies on complete, accessible, and reusable research artifacts, yet artifact evaluation remains largely manual and difficult to scale. This emerging results paper explores an agentic approach for assessing replication package quality by translating open-science guidelines into machine-verifiable criteria. We consolidate 380 requirements from 34 sources into 51 reproducibility criteria, of which 31 are operationalized for automated artifact-based evaluation. Based on these criteria, we implement a multi-agent prototype that automatically inspects replication packages and produces evidence-grounded improvement reports. A preliminary evaluation on five replication packages shows high inter-run consistency of 91.4% and 75.4% correctness, through micro-averaged agreement with a manual baseline. The agent performs best on structural criteria such as code, environment, and artifact availability, but struggles with qualitative or mixed-method studies. A pilot survey with seven software engineering researchers indicates well perceived usefulness and adoption potential, while revealing cognitive load in the human-in-the-loop planning step. Overall, these emerging results indicate that agentic research artifact evaluation has the potential to support authors and reviewers by automating selected routine checks. Technical University of Munich, Germany and https://github.com/CallMeSwarley [email protected]://orcid.org/0009-0003-3831-1007 fortiss, Germany and Blekinge Institute of Technology, Sweden and https://angermeir.me [email protected]://orcid.org/0000-0001-7903-8236 {CCSXML} ¡ccs2012¿ ¡concept¿ ¡concept_id¿10002944.10011123.10010912¡/concept_id¿ ¡concept_desc¿General and reference Empirical studies¡/concept_desc¿ ¡concept_significance¿500¡/concept_significance¿ ¡/concept¿ ¡concept¿ ¡concept_id¿10011007.10011074.10011099.10011693¡/concept_id¿ ¡concept_desc¿Software and its engineering Empirical software validation¡/concept_desc¿ ¡concept_significance¿500¡/concept_significance¿ ¡/concept¿ ¡/ccs2012¿ ccs General and reference Empirical studies ccs Software and its engineering Empirical software validation keywords reproducibility, software engineering, research artifacts, agentic systems, multi-agent systems, evaluation framework, research infrastructure † † runningtitle: An Agentic Approach Towards Replication Package Quality Evaluation † † runningauthor: M. Amougou et al. † † copyright: Maximilian Alexander Amougou Mbida and Florian Angermeir 1 Introduction Reproducibility is a fundamental prerequisite for cumulative scientific progress, enabling independent verification of findings and fostering confidence in published research [ 17 , 6 ] . In large parts of empirical software engineering, reproducibility depends not only on textual reporting, but also on the availability and quality of accompanying research artifacts such as source code, datasets, execution scripts, configuration files, and experimental environments [ 24 ] . To enhance transparent research practices, conferences and publishers have introduced artifact-evaluation tracks, open science guidelines, and reporting standards [ 18 , 2 ] . Despite these efforts, research artifacts frequently suffer from a plethora of issues, including incomplete source code [ 16 , 12 ] , missing dependencies [ 23 , 13 ] , and fragmented documentation [ 11 ] . Consequently, verifying whether these packages are sufficiently complete, accessible, and reusable remains a manual, time-consuming process that scales poorly and is subject to interpretation [ 25 , 24 , 9 , 21 ] . While existing guidelines outline what researchers should report, they provide limited support for automatically assessing the quality of heterogeneous artifact types against predefined criteria representing open science best practices. As a result, artifact evaluation continues to impose substantial effort on reviewers, slows down review processes, and creates barriers for large-scale reproducibility assessments. This challenge is particularly pronounced in contemporary software engineering research, where replication packages lack standardization and often bundle a mix of source code, raw datasets, and configuration files across various directories and formats. In this paper, we explore whether an agentic approach can support replication package quality evaluation by operationalizing open-science guidelines into machine-actionable verification procedures. We propose an agent-based approach that autonomously retrieves, structures, and evaluates research artifacts against reproducibility criteria, enabling a scalable assessment of research artifacts quality as a proxy to the potential for reproducibility. We hope that our contribution in the long-term can support authors in improving their research artifact quality even before artifact review and to consequently reduce review overhead in artifact review tracks. To investigate the feasibility of this idea, we implemented a prototype based on a multi-agent architecture and conducted a pilot survey. Our preliminary evaluation indicates promising results. The approach achieves an inter-run consistency of 91.4% and a 75.4% micro-averaged agreement with manual baseline. While still preliminary, these results strengthen our confidence in the potential and long-term value of the proposed approach. The main contributions of this paper are (1) an operationalization of open-science guidelines into a set of machine-verifiable criteria for automated artifact assessment, (2) a novel agentic approach for assessing the reproducibility potential of software engineering research artifacts, and (3) preliminary empirical evidence from a pilot survey that demonstrates the potential and limitations of our approach. 2 Related Work We structure existing related work into challenges with manual and static artifact evaluation, and autonomous reproduction systems. Challenges and Automated Artifact Evaluation. Achieving sustainable reproducibility is hindered by the post-publication fragility of long-term archiving [ 24 ] and a pre-publication reliance on manual human labor during peer review. This constraint leads to inconsistent evaluation depth and limits scalability, driving the necessity for automated review assistance. While earlier automation focuses on static analysis of manuscripts, such as Metacheck [ 8 ] , such solutions can be limited by the lacking of semantic analysis depth of research artefacts. Our approach resolves this through fully automated, end-to-end evaluation of both the artifact content and the structural alignment. Agentic Reproducibility Benchmarks and Frameworks. Recent benchmarks quantify LLM capabilities in scientific reproduction, typically reporting low success rates (20% to 40%) that highlight domain complexity. Evaluation paradigms target either execution and consistency checks (such as dependency installation in CORE-Bench [ 21 ] and paper-to-claim alignment in REPRO-BENCH [ 14 ] ), or the harder task of generating reproduction code from scratch, as evaluated by PaperBench [ 22 ] and LMR-BENCH [ 26 ] . To tackle these complexities, several architectures employ specialized reasoning pipelines. For instance, AutoReproduce [ 27 ] and ResearchCodeAgent [ 10 ] leverage paper lineages and execution feedback to iteratively refine scripts, while SciRep/SciConv [ 4 , 5 ] automate Docker-based setups. Closest to our work, ArtifactCopilot [ 25 ] maps unstructured documentation to dependency graphs for end-to-end ACM badging [ 9 ] . However, most of the agentic approaches assume well-structured repositories, complete documentation, or explicit instructions. In contrast, our framework focuses on evaluating the reproducibility potential of heterogeneous, inconsistently structured research artifacts where documentation may be partial with the goal of enhancing artifact quality. 3 Methodology To guide the design, implementation, and evaluation of our approach, we organized our methodology around three central research questions: RQ1: How can an agentic approach assess the potential for reproducibility given a research artifact set? RQ2: What are the potential and limitations of an agentic approach to assess the quality of a replication package? RQ3: How useful is the agentic approach for stakeholders assessing replication package quality? Reproducibility Criteria To establish robust reproducibility criteria, we analyzed official guidelines and reviewer resources from top-tier (CORE A/A* [ 3 ] ) computer science conferences featuring artifact evaluation tracks or explicit checklists (e.g., ICSE, OSDI, ICML, RE). This was supplemented by open science policies from major publishers, leading empirical software engineering journals (EMSE, IST, IEEE Access [ 20 ] ), and community-driven best practices. Extraction Process. We categorized the source documents into four groups: (1) conferences with artifact badges, (2) checklist-only venues, (3) journals and publishers, and (4) general guidelines. Verbatim criteria were extracted into a centralized