Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios

Ruida Hu 1 , Xinchen Wang 1 , Chao Peng 2 , Cuiyun Gao 1† , David Lo 3 · 2026

Read Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios by Ruida Hu 1 , Xinchen Wang 1 , Chao Peng 2 , Cuiyun Gao 1† , David Lo 3 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06742.txt.

Introduction

Hu and four colleagues test whether language models can build command-line tools from scratch. Older benchmarks hand models structural scaffolds, reducing the job to file filling, or grade with rigid white-box unit tests that force specific internals. Their CLI-Tool-Bench gives agents an empty workspace across 94 real repositories and grades finished tools by black-box differential testing against human-written oracles. Seven leading models were tried, and the best reached under 44 percent overall success, with agents tending toward monolithic code and extra tokens buying no better results.

Catalog record

CREATORRuida Hu 1 , Xinchen Wang 1 , Chao Peng 2 , Cuiyun Gao 1† , David Lo 3
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55940
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2604.06742

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 Thanks: † Corresponding author. CCS: Software and its engineering Software verification and validation Ruida Hu 1 , Xinchen Wang 1 , Chao Peng 2 , Cuiyun Gao 1† , David Lo 3 Abstract . The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to architect and deliver complete, runnable software systems entirely from scratch. However, existing benchmarks fail to adequately assess this promising 0-to-1 software generation capability due to two fundamental limitations. First, they predominantly rely on predefined structural scaffolds, regarding the task as mere file-filling and failing to evaluate the LLM’s crucial ability in repository structure planning. Second, they heavily depend on rigid white-box unit testing, which forces agents to conform to specific internal implementations and lacks scalable, end-to-end behavioral validation from a user-centric perspective. To bridge this gap, we introduce CLI-Tool-Bench, a novel, structure-agnostic benchmark designed to evaluate the ground-up generation of Command-Line Interface (CLI) tools. Powered by a black-box differential testing framework, CLI-Tool-Bench comprises 100 high-quality, real-world repositories spanning diverse difficulty levels, programming languages, and application domains. For each repository, our automated pipeline synthesizes a robust suite of end-to-end test cases. We evaluate the agent-generated software by executing it in isolated sandboxes and comparing its system-level side effects and terminal outputs against human-written oracles using a rigorous, multi-tiered equivalence metric. Leveraging CLI-Tool-Bench, we conduct an extensive evaluation of seven state-of-the-art LLMs deployed within prominent agent frameworks. Our experiments reveal that top-tier models achieve less than 43% overall success, highlighting that 0-to-1 generation remains a highly challenging frontier. Furthermore, we discover that higher token consumption does not necessarily yield better performance, and agents exhibit a strong tendency to generate monolithic code structures. 1. Introduction The advent of Large Language Models (LLMs) has catalyzed a paradigm shift in automated software engineering ( 1 ; 16 ; 5 ; 11 ) . Moving beyond simple code completion, recent advancements have spurred the development of autonomous LLM-based agents ( 3 ; 4 ; 14 ; 24 ) capable of tackling complex programming tasks. This evolution has given rise to the era of “Vibe Coding” or intent-driven development ( 21 ; 15 ; 18 ) , where users, ranging from professional developers to non-technical individuals, can generate functional software simply by expressing their high-level requirements in natural language. In this new paradigm, autonomous agents are expected to take a high-level natural language requirement and generate a complete, runnable software repository entirely from scratch. To measure the capabilities of these agents, robust evaluation benchmarks are indispensable. However, existing benchmarks fall short of evaluating the true potential of agents in real-world software creation. Traditional benchmarks, such as HumanEval ( 6 ) and MBPP ( 2 ) , are confined to function-level or snippet-level generation. While recent efforts like SWE-bench ( 12 ) have elevated the evaluation to the repository level, they predominantly focus on software maintenance tasks—such as resolving issues or adding features to existing codebases—rather than the creation of software from scratch. More recently, attempts like NL2Repo-Bench ( 7 ) have explored repository-level generation from scratch; however, their evaluation methodologies reveal critical limitations that hinder the accurate assessment of modern LLM agents. Specifically, we identify three major challenges in the current evaluation landscape: Challenge 1: Reliance on Predefined Repository Structures. Despite rapid advancements, evaluating the true 0-to-1 generation capabilities of LLM agents remains an open challenge. Existing repository-level benchmarks predominantly focus on issue resolution or feature addition within already established repositories ( 12 ) . Even in generation-focused benchmarks, the evaluation heavily relies on a fixed, predefined repository structure ( 7 ) . These benchmarks typically provide agents with pre-built file skeletons and directory scaffolds, reducing the complex task of software generation to mere code-filling. This structure-dependent paradigm fundamentally bypasses a critical step in software creation: repository structure planning. In real-world 0-to-1 development, developers must autonomously decide how to organize directories, modularize files, and manage dependency configurations. By constraining agents to predefined structures, current benchmarks fail to assess whether LLMs can independently plan and construct a coherent repository from scratch. Challenge 2: Absence of End-to-End Black-Box Testing. Furthermore, existing evaluations heavily rely on white-box unit testing. These tests are tightly coupled with the internal implementation details of the software, forcing the generated code to conform to specific function signatures or class definitions. However, from a user-centric perspective, software utilities, such as Command-Line Interface (CLI) tools, are always consumed as black boxes. Users care about whether the tool correctly parses command-line arguments, produces the expected terminal outputs, and executes the correct system-level side effects (e.g., modifying the file system), regardless of how the internal code is structured. The reliance on rigid white-box testing not only stifles the structural autonomy of LLMs but also fails to provide a realistic, end-to-end validation of the software’s functional correctness. To address these limitations, we introduce CLI-Tool-Bench, a novel benchmark specifically designed to evaluate the 0-to-1 generation of CLI tools. CLI-Tool-Bench shifts the evaluation paradigm from structure-dependent white-box testing to structure-agnostic black-box differential testing. We curate a high-quality dataset of 100 real-world CLI repositories across three programming languages (Python, JavaScript, Go) and complexity levels. To overcome the reliance on predefined structures (Challenge 1), the agent is provided only with a natural language requirement and an empty workspace. This unconstrained setting forces the LLM to autonomously handle repository structure planning, dependency management, and logic implementation from scratch. To provide realistic, end-to-end validation (Challenge 2), we evaluate the generated tool purely from a user-centric perspective. We execute the tool in an isolated sandbox and compare its terminal outputs and system-level side effects (e.g., file system state changes) against a human-written oracle. To ensure a fair assessment, we propose a multi-tiered equivalence metric—encompassing Execution, Exact, Fuzzy, and Semantic Match—to accurately gauge behavioral correctness without penalizing the agent’s architectural diversity. In summary, the main contributions of this paper are as follows: \bullet We introduce CLI-Tool-Bench, the first benchmark for end-to-end software generation. It challenges agents to build functional CLI tools from scratch, granting them complete autonomy without predefined scaffolds. \bullet We propose an automated pipeline for repository-level generation. It utilizes black-box differential testing in isolated Docker environments to rigorously assess execution reliability, behavioral equivalence, and system-level side effects. \bullet We evaluate advanced LLMs and agent frameworks on CLI-Tool-Bench, revealing their great struggles with autonomous system-level generation. We also uncover critical behavioral patterns, such as monolithic design preferences and infinite generation loops. 2. Construction and Evaluation Pipeline Figure 1. Overview of the CLI-Tool-Bench framework. To evaluate the 0-to-1 software generation capabilities of LLM agents, we propose an automated benchmark construction and evaluation pipeline. As illustrated in Figure 1 , it consists of three core modules: (1) Repository Curation, (2) Schema-Guided Task Synthesis, and (3) Black-Box Differential Evaluation. 2.1. Repository Curation To construct a high-quality and representative dataset, we target three mainstream languages widely adopted for CLI development, including Python, JavaScript, and Golang ( 13 ) . To ensure the reliability of the oracle repositories serving as ground truth in

Read the complete text