AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents

Bhanu Prakash Vangala, Ali Adibifar, Tanu Malik, Ashish Gehani · 2025

Read AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents by Bhanu Prakash Vangala, Ali Adibifar, Tanu Malik, Ashish Gehani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22387.txt.

Introduction

Bhanu Prakash Vangala, Ali Adibifar, Ashish Gehani, and Tanu Malik test whether code written by AI agents runs. They took three leading coding agents, Claude Code, OpenAI Codex, and Gemini, and had them build 300 projects from 100 standardized prompts in Python, JavaScript, and Java. Each project was then launched in a clean environment with only operating system packages plus whatever dependencies the model itself specified.

Only about sixty-eight percent of the projects executed out of the box. Python fared best at roughly eighty-nine percent, while Java managed just forty-four percent. The team separates claimed, working, and runtime dependencies, and found actual runtime needs expanded more than thirteenfold beyond what models declared. Hidden dependencies, they conclude, are the reproducibility gap.

Catalog record

CREATORBhanu Prakash Vangala, Ali Adibifar, Tanu Malik, Ashish Gehani
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22387

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents Bhanu Prakash Vangala, Ali Adibifar, Tanu Malik, Ashish Gehani Abstract The rise of Large Language Models (LLMs) as coding agents promises to accelerate software development, but their impact on generated code reproducibility remains largely unexplored. This paper presents an empirical study investigating whether LLM-generated code can be executed successfully in a clean environment with only OS packages and using only the dependencies that the model specifies. We evaluate three state-of-the-art LLM coding agents (Claude Code, OpenAI Codex, and Gemini) across 300 projects generated from 100 standardized prompts in Python, JavaScript, and Java. We introduce a three-layer dependency framework (distinguishing between claimed, working, and runtime dependencies) to quantify execution reproducibility. Our results show that only 68.3% of projects execute out-of-the-box, with substantial variation across languages (Python 89.2%, Java 44.0%). We also find a 13.5× average expansion from declared to actual runtime dependencies, revealing significant hidden dependencies. Introduction Reproducibility forms the cornerstone of computational science. When researchers share code, others must be able to execute it, verify results, and build upon the work ( 7 ) . This fundamental principle enables scientific progress, validates discoveries, and ensures reliability in production systems. Yet as we increasingly rely on Large Language Models (LLMs) to generate code, a troubling question emerges: is AI-generated code reproducible? The computational research community already faces a reproducibility crisis. Studies show that a significant portion of published AI/ML research cannot be reproduced due to missing dependencies, incomplete environment specifications, and undocumented requirements ( 16 ; 17 ; 9 ; 8 ) . Now, tools like Claude Code, OpenAI Codex, and Gemini Code Assist promise to accelerate development by generating complete projects, entire applications with multiple files, configurations, and dependency specifications. But if these AI systems inherit or amplify reproducibility problems, we risk compounding the crisis rather than solving it. Consider what reproducibility means for AI-generated code. A developer asks an LLM to create a complete sentiment analysis web application. The AI generates an impressive project: backend API files, frontend code, and a requirements.txt listing dependencies. For this code to be reproducible, another developer should be able to take these files and specifications, set up a clean environment, install the listed dependencies, and run the application. But in practice, execution fails in 31.7% of cases. While missing dependencies like ImportError: No module named nltk contribute to some failures (10.5%), the majority stem from fundamental code generation errors: malformed syntax, incorrect file paths, uninitialized variables, and structural issues. After manually debugging, a process that we found takes 15 minutes on average, some projects eventually run. However, even successful projects hide complexity - requiring an average of 37 packages at runtime instead of the 3 typically declared. The code is not reproducible as generated. This reproducibility failure represents a fundamental gap in how we evaluate AI coding agents. Current benchmarks like HumanEval ( 3 ) and MBPP ( 2 ) test functional correctness but assume reproducible environments already exist, a recognized bottleneck that obscures key failure modes related to environment configuration ( 24 ; 12 ) . They ask, does this sorting algorithm work correctly? without first asking, can anyone actually run this code? This misses a critical insight: code cannot be correct if it cannot be reproduced. Reproducibility is not a secondary concern; it is a prerequisite for verification, collaboration, and deployment. To measure reproducibility in AI-generated code, we introduce Executable Reliability: the likelihood that a project executes successfully in a clean environment using only the dependencies and instructions the AI provides. This metric directly captures reproducibility; can others reproduce the execution without additional knowledge or debugging? We develop a three-layer framework to understand reproducibility failures: claimed dependencies (what the AI specifies), working dependencies (what’s actually needed for reproduction including transitive), and runtime dependencies after executing (the complete transitive closure). The gaps between these layers reveal where reproducibility breaks down. We conducted a systematic reproducibility study of 300 complete projects generated by three leading state of the art LLM agents ( 1 ; 14 ; 6 ) . Each AI received identical prompts explicitly requesting reproducible code with complete dependency specifications. We attempted to reproduce each project in clean environments with only OS packages installed and none other than them, documenting every failure and missing requirement. This mimics the exact challenge faced by researchers trying to reproduce published code or developers attempting to use AI-generated projects. Our methodology treats reproducibility as the primary concern: can the generated code be executed by others using only what the AI provides? Our findings reveal that AI-generated code faces a severe reproducibility crisis. Only 68.3% of projects are reproducible without manual intervention nearly one-third fail to execute as specified. The reproducibility rates vary dramatically by language: Python achieves 89.2% reproducibility, JavaScript 61.9%, and Java just 44.0%. This variation reflects how different ecosystems handle dependency specifications ( 5 ; 13 ) and transitive requirements. Most striking is the runtime dependency explosion: projects claiming 3 dependencies actually require an average of 37 packages, and 13.5× gap between specified and actual requirements. This hidden complexity makes reproduction nearly impossible without extensive debugging. These results demonstrate that current LLMs generate code that appears complete but lacks the specifications necessary for reproducibility. The implications extend beyond convenience. For scientific computing, non-reproducible code undermines research validity. For software development, it creates hidden technical debt ( 18 ) . For education, it teaches poor practices. By quantifying these reproducibility failures and identifying their patterns, we provide both a diagnosis of the current crisis and a roadmap for creating AI systems that generate truly reproducible code. The challenge is not just to generate code that works on the developer’s machine, but code that others can reliably execute, the foundation of computational reproducibility. Background and Related Work Reproducibility Crisis in AI-Generated Code The reproducibility crisis in AI research ( 7 ) now extends to the tools meant to address it. While computational research should enable perfect reproducibility ( 15 ) , LLM-generated code introduces new failure modes. Studies show LLMs produce inconsistent outputs ( 23 ) and unreliable automation ( 20 ) . Our work reveals a more fundamental issue: even when LLMs generate complete projects with multiple files and configurations, they fail to specify dependencies needed for reproducible execution. Benchmarks Missing Reproducibility Current benchmarks like HumanEval and MBPP ( 3 ) evaluate functional correctness assuming reproducible environments exist. LiveCodeBench prevents memorization but still provides complete environments ( 10 ) . These benchmarks test whether code is correct, not whether it’s reproducible. Our work evaluates complete projects not snippets asking whether they can execute at all using only what LLMs provide, making reproducibility the primary metric. Dependency Complexity and Reproducibility Software reproducibility requires complete dependency specification. JavaScript packages average 79 transitive dependencies ( 5 ) . While tools like ReproZip ( 4 ) and SciUnit ( 22 ) capture execution environments for reproducibility, LLMs lack this awareness. They generate syntactically correct code but fail to specify the dependency closure required for reproduction. Gap in Reproducibility Evaluation To our knowledge, no prior work systematically evaluates whether LLM-generated projects are reproducible. Existing research focuses on code correctness or captures environments post-hoc.

Read the complete text