Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches
S. Mehtab Hussain Shah, Frank Hopfgartner, Arnim Bleier · 2026
Read Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches by S. Mehtab Hussain Shah, Frank Hopfgartner, Arnim Bleier (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-08561.txt.
Description
Reproducing a published analysis should be as simple as rerunning code and data, yet missing packages, fragile paths, and version clashes often break it. Syed Mehtab Hussain Shah and two co-authors test whether language models and agents can diagnose and repair such failures automatically. The testbed drew on five social-science studies in R that already reproduced cleanly.
The team then planted realistic failures, from simple glitches to missing logic, and both repair workflows ran in fresh Docker sandboxes. Prompt-based repair succeeded in 31 to 79 percent of runs, depending on context and complexity. Agent-based repair, which inspects files and reruns analyses itself, reached 69 to 96 percent. Readers get evidence that agents can shoulder much of the reproduction burden.
Catalog record
| CREATOR | S. Mehtab Hussain Shah, Frank Hopfgartner, Arnim Bleier |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-08561