Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches

S. Mehtab Hussain Shah, Frank Hopfgartner, Arnim Bleier · 2026

Read Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches by S. Mehtab Hussain Shah, Frank Hopfgartner, Arnim Bleier (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-08561.txt.

Description

Reproducing a published analysis should be as simple as rerunning code and data, yet missing packages, fragile paths, and version clashes often break it. Syed Mehtab Hussain Shah and two co-authors test whether language models and agents can diagnose and repair such failures automatically. The testbed drew on five social-science studies in R that already reproduced cleanly.

The team then planted realistic failures, from simple glitches to missing logic, and both repair workflows ran in fresh Docker sandboxes. Prompt-based repair succeeded in 31 to 79 percent of runs, depending on context and complexity. Agent-based repair, which inspects files and reruns analyses itself, reached 69 to 96 percent. Readers get evidence that agents can shoulder much of the reproduction burden.

Catalog record

CREATORS. Mehtab Hussain Shah, Frank Hopfgartner, Arnim Bleier
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-08561

Browse the full library · About the Tower of Babel