RepoZero: Can LLMs Generate a Code Repository from Scratch?

Zhaoxi Zhang, Yiming Xu, Jiahui Liang, Weikang Li, Xiaoshuai Chen, Liwei Qian, Xin Pei, Jizhou Huang, Rui Sun, Yunfang Wu · 2026

Read RepoZero: Can LLMs Generate a Code Repository from Scratch? by Zhaoxi Zhang, Yiming Xu, Jiahui Liang, Weikang Li, Xiaoshuai Chen, Liwei Qian, Xin Pei, Jizhou Huang, Rui Sun, Yunfang Wu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-07122.txt.

Introduction

Zhaoxi Zhang and nine co-authors ask whether language models can build a whole code repository from scratch. RepoZero, their benchmark, replaces patch-style editing tasks with full reproduction. Given only an API specification, an agent must reimplement a repository matching the original's behavior. Output equivalence allows strict, automatic checking.

Cross-language rules and a sandbox limit cheating and data leakage. Their companion framework grows tests iteratively and refines code from errors, enabling test-time scaling. Even the strongest agents pass only a limited share of tasks, exposing a wide gap between current coding models and real software development needs.

Catalog record

CREATORZhaoxi Zhang, Yiming Xu, Jiahui Liang, Weikang Li, Xiaoshuai Chen, Liwei Qian, Xin Pei, Jizhou Huang, Rui Sun, Yunfang Wu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.07122

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RepoZero: Can LLMs Generate a Code Repository from Scratch? Zhaoxi Zhang, Yiming Xu, Jiahui Liang, Weikang Li, Xiaoshuai Chen, Liwei Qian, Xin Pei, Jizhou Huang, Rui Sun, Yunfang Wu Abstract Large Language Models (LLMs) have recently shown remarkable progress in code generation, yet their ability to construct complete software repositories from scratch remains poorly understood. A fundamental bottleneck is the lack of verifiable and scalable evaluation: existing benchmarks either focus on patch-based editing or rely on human or LLM-based judgments, which introduce bias and limit reproducibility. In this work, we present RepoZero, the first benchmark that enables fully automated, execution-based verification of repository-level generation from scratch. Our key idea is to reformulate generation as repository reproduction: given only API specifications, an agent must re-implement an entire repository such that its behavior matches the original implementation. This design allows for strict black-box validation via output equivalence, while naturally supporting large-scale construction by reusing existing open-source repositories. To further mitigate data leakage and shortcut solutions, we introduce cross-language constraints and a sandboxed evaluation protocol. Building on this benchmark, we propose an Agentic Code-Test Evolution (ACE) framework that performs iterative test generation and error-driven refinement, enabling effective test-time scaling for repository-level synthesis. Extensive experiments across multiple state-of-the-art LLMs and agent frameworks reveal that even the strongest LLM agents achieve only limited pass rates (30% - 55%), exposing a substantial gap between current capabilities and real-world software development requirements. Our results establish RepoZero as a challenging, scalable, and reliable testbed for end-to-end code generation, and highlight self-verification via test generation as a critical direction for advancing LLM-based coding agents. Code is available at https://github.com/JesseZZZZZ/RepoZero . Table 1: Comparison with existing coding benchmarks on multiple dimensions. Repository (generating a repository rather than a single code snippet), Multi-lang. (multiple programming languages), From-scratch Gen. (generation with no pre-given code), Execution (verifying the executed results), Test Suites (test cases rather than LLM-as-judge), Scalability (automatic pipeline rather than human-required), and Leakage Defense (preventing data leakage). Benchmark Repository Multi-lang. From-scratch Gen. Execution Test Suites Scalability Leakage Defense HumanEval ( 2 ) ✗ ✗ ✗ ✓ ✓ ✗ ✓ SWE-bench ( 12 ) ✓ ✗ ✗ ✗ ✓ ✗ ✗ Multi-SWE-bench ( 34 ) ✓ ✓ ✗ ✗ ✓ ✗ ✗ FEA-bench ( 13 ) ✓ ✗ ✗ ✗ ✓ ✗ ✗ EvoCodeBench ( 37 ) ✓ ✗ ✓ ✗ ✓ ✗ ✗ CodeS ( 35 ) ✓ ✗ ✓ ✗ ✗ ✗ ✗ E2EDevBench ( 15 ) ✓ ✗ ✓ ✗ ✗ ✗ ✓ NL2Repo ( 7 ) ✓ ✗ ✓ ✓ ✓ ✗ ✗ RepoZero (Ours) ✓ ✓ ✓ ✓ ✓ ✓ ✓ 1 Introduction Recent advances in LLM-based coding have drawn increasing attention toward fully autonomous end-to-end software development. State-of-the-art large language models ( 14 ; 30 ; 25 ; 29 ; 36 ) have been integrated into coding agents ( 27 ; 9 ; 38 ) capable of completing complex programming tasks with minimal or no human intervention. From the perspective of benchmark difficulty, coding tasks can generally be categorized along two orthogonal dimensions. The first dimension concerns the scope of code modification, including: (1) single-file coding and (2) repository-level coding. The second dimension concerns the nature of the programming task itself, including: (1) debugging, (2) incremental development, and (3) generation from scratch. Current coding agents have largely mastered single-file editing. However, evaluating repository-level modifications remains a burgeoning frontier. Benchmarks such as SWE-bench ( 12 ) , Multi-SWE-bench ( 34 ) , and FEA-bench ( 13 ) have pioneered the assessment of repo-level edits and incremental development in real-world scenarios. While these benchmarks leverage patch-based verification and unit testing, the domain of from-scratch repository generation remains relatively unexplored. A primary obstacle is the scarcity of pre-existing unit tests available online for newly generated projects. Consequently, current benchmarks for from-scratch generation, such as CodeS ( 35 ) , EvoCodeBench ( 37 ) , and NL2RepoBench ( 8 ) , predominantly rely on human-in-the-loop or LLM-as-a-judge frameworks. These methods, however, introduce subjective biases from both human evaluators and language models, leading to potentially unreliable evaluation outcomes. Data leakage presents another formidable challenge for repository-level coding benchmarks. Since modern large language models (LLMs) are extensively pre-trained on massive GitHub corpora ( 36 ; 29 ) , it is increasingly difficult to discern whether their performance stems from genuine reasoning capabilities or mere rote memorization. While some studies ( 37 ) attempt to mitigate this by utilizing rare or recently published repositories, this approach inherently constrains the dataset scale, thereby precluding large-scale evaluation. Furthermore, such "clean" datasets are subject to rapid contamination post-publication, necessitating frequent and labor-intensive updates to maintain their integrity. To address these challenges, we propose RepoZero , the first verifiable benchmark specifically designed to evaluate the from-scratch repository-level generation capabilities of LLM agents. RepoZero achieves rigorous verification by reformulating repository-level generation as a repository reproduction task. Specifically, for a target repository (e.g., a pip-installed Python package), we first employ an LLM to generate test files that invoke the repository’s APIs; for each test file, the LLM generates several test cases, which are then filtered to retain only successful test cases as ground truth. An agent is then tasked with generating an entire target repository that replicates the source repository’s functionality and API signatures. Evaluation is performed by executing the generated target repository against the saved test cases and comparing the outputs with those of the source repository. To mitigate data leakage, we mandate cross-language synthesis–requiring agents to implement the repository in a different programming language–and prohibit the use of external site packages. Furthermore, we introduce an Agentic Code-Test Evolution (ACE) workflow for test-time scaling. Unlike existing frameworks, where the lack of a verifiable environment makes retrieving ground-truth outputs for LLM-generated tests impossible, RepoZero leverages the source repository as an oracle, providing a deterministic and automated gold standard for evaluation. In summary, our primary contributions are as follows: 1. Introduction of RepoZero : We establish the first verifiable and highly scalable benchmark tailored for repository generation from scratch, bridging the evaluation gap in existing repo-level coding tasks. 2. Test-Time Scaling Framework : We propose a novel agentic framework centered on test-time scaling, which significantly enhances the success rate of complex, repository-level synthesis. 3. Comprehensive Benchmarking : We conduct an extensive evaluation of state-of-the-art models and agentic scaffolds, providing critical insights and a robust foundation for future research in autonomous software engineering. 2 Benchmark: RepoZero 2.1 Statistics of RepoZero RepoZero consists of two subsets: RepoZero-Py2JS , where the source repositories are implemented in Python, and the agent is required to reimplement them in JavaScript, and RepoZero-C2Rust , where the source repositories are written in C/C++ and the agent is tasked with reimplementing them in Rust. As illustrated in Fig. 1 , RepoZero covers a diverse range of repository categories. All repositories are manually curated and further preprocessed following the workflow described in Sec. 3 . We employ LLMs to estimate the difficulty of each sample under several rubrics: Lines of Code (LOC), API counts, and input/output complexity. We apply majority voting (Claude-4.6-Opus, GLM-5, and DeepSeek-V3.2) to obtain the final estimation. The datasets are accordingly partitioned into Easy , Medium , and Hard subsets. From the curated repositories, we construct 400

Read the complete text