From Defects to Demands: A Unified, Iterative, and Heuristically Guided LLM-Based Framework for Automated Software Repair and Requirement Realization
Alex(Baoyuan) Liu, Vivian(Zirong) Chi · 2024
Read From Defects to Demands: A Unified, Iterative, and Heuristically Guided LLM-Based Framework for Automated Software Repair and Requirement Realization by Alex(Baoyuan) Liu, Vivian(Zirong) Chi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05098.txt.
Introduction
Two authors set out a loop frame that lets language models fix code bugs and build asked traits in one flow. Their loop weds the model with hard proof checks, test-led builds, and step-wise plan hints. Each round grades the code, finds the fail, and steers the next patch by that grade. The frame is cast for both repair shops and fresh ask builds.
On a famed code bench, their sums claim a jump of near thirty-nine points over the top past score. Such a jump, if it holds, would mark a leap in auto code worth. The pair frame the gain as proof that proof-led loops beat raw model shots. Code teams are urged to vet the claim on their own benches.
Catalog record
| CREATOR | Alex(Baoyuan) Liu, Vivian(Zirong) Chi |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-05098
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Defects to Demands: A Unified, Iterative, and Heuristically Guided LLM-Based Framework for Automated Software Repair and Requirement Realization Alex(Baoyuan) Liu, Vivian(Zirong) Chi Abstract This manuscript represents the dawn of a new epoch in the symbiosis of artificial intelligence and software engineering, heralding a paradigm shift that places machines indisputably at the pinnacle of human capability. For the first time in recorded history, we present a rigorously formalized, iterative, and heuristically optimized methodology proving—without ambiguity—that AI can fully supplant human programmers in all aspects of code construction, refinement, and evolution. Our integrated approach, which seamlessly fuses state-of-the-art Large Language Models with advanced formal verification, dynamic test-driven development, and incremental architectural guidance, achieves an unprecedented milestone: a 38.6% improvement over the current top performer’s accuracy (48.33%) on the industry-standard SWE-bench benchmark. This surges performance to an apex heretofore considered unattainable, effectively signaling the obsolescence of human-exclusive coding and the ascendancy of autonomous AI-driven software innovation. 1 Introduction As software systems evolve, developers face a dual challenge: maintaining correctness by fixing bugs and continuously adapting functionality to meet new user demands. Traditional software engineering processes rely heavily on human developers to interpret requirements, fix errors, and ensure correctness against specifications. With the advancement of Large Language Models (LLMs) adept at code generation, the opportunity arises to shift portions of these responsibilities onto machine-driven processes. However, simply prompting an LLM to solve a complex programming task—be it eliminating a subtle bug or implementing a new feature—often falls short. Complex codebases exceed the model’s context window, specification details are not always fully captured in a single prompt, and correctness requires iterative refinement guided by tests, analysis, and verification. This paper proposes a holistic, iterative framework that enables an LLM to evolve a codebase from an initial, potentially buggy state to one that satisfies not only pre-existing correctness criteria but also newly introduced feature demands. Key contributions include: 1. Unified Framework for Bug-to-Demand Resolution: We present a method by which the LLM iteratively refines code, starting from an imperfect state (with known or unknown bugs) and incrementally adjusting the codebase to meet a set of evolving functional and non-functional requirements introduced over time. 2. Test-Driven and Verification-Guided Iteration: By integrating test-driven development (TDD), logging, incremental formal verification, and static analysis, our approach provides multi-channel feedback at every iteration, ensuring that each step either moves closer to a correct and fully realized solution or identifies actionable next steps. 3. Heuristic Search and Context Management: We introduce a heuristic search manager that ranks candidate changes and leverages parallelization and context-aware code retrieval to handle large codebases, limited LLM context windows, and complexity introduced by incremental demands. 4. Formal Guarantees of Convergence: Building on classical enumerability and verification arguments, we provide a detailed theoretical demonstration that, under reasonable assumptions, the iterative process converges to a state in which all specified demands and correctness criteria are met in finite time. In essence, this work envisions an LLM-driven automated software engineer: not only capable of fixing existing defects but also extending and evolving the software to fulfill new requirements as they emerge. 2 Related Work Existing studies in automated program repair (APR) and code synthesis have produced promising results, but typically focus on isolated bug fixes [ 4 , 5 , 6 ] . Research on synthesis from specifications has shown that iterative refinement can be guided by formal methods [ 7 ] or test-based oracles [ 3 ] . Test-driven development and regression testing techniques have informed various APR methods [ 8 , 9 ] . Large Language Models trained on extensive corpora of code have been leveraged as coding assistants, but their raw capabilities often falter in complex, iterative, and context-rich tasks [ 10 , 11 ] . Our approach extends prior work by integrating multi-faceted verification and feedback mechanisms with LLMs, and by addressing both defect repair and incremental demand implementation, thus bridging the gap between automated code refinement and agile requirement realization. 3 Foundational Concepts and Assumptions Achieving a fully automated, iterative refinement of a complex software codebase—extending from the correction of subtle defects to the realization of newly introduced requirements—demands a firm mathematical foundation. In this chapter, we establish a rigorous and comprehensive theoretical framework that captures the structure of candidate solutions, the evolving specification, and the interplay of testing, verification, and heuristic guidance. By doing so, we lay the groundwork for a methodology that aspires to the highest standards of scientific rigor and technical completeness. 3.1 Representation of Code and Hypothesis Space Consider a codebase C as a structured entity encompassing functions, classes, modules, and data definitions. Let C_{0} be an initial code version, which may exhibit known or unknown defects. We define a hypothesis space \mathcal{H} , typically countably infinite, representing all candidate code variants reachable from C_{0} through a finite sequence of edits. Formally: \mathcal{H}=\{C^{(i)}\mid i\in\mathbb{N}\}. Each element C^{(i)}\in\mathcal{H} is associated with a semantic interpretation f_{C^{(i)}}:D\to R , mapping inputs d\in D to outputs r\in R . The specification \varphi , which may evolve over time, imposes constraints on the permissible behaviors of f_{C} . Because we aim to not only fix existing bugs but also to satisfy newly introduced demands, \varphi is itself dynamic. Suppose \varphi consists of L logical clauses \{\varphi_{1},\varphi_{2},\dots,\varphi_{L}\} . Each \varphi_{l} represents a particular correctness, performance, or architectural requirement. We measure compliance via: \mu_{C,\varphi}=\frac{1}{L}\sum_{l=1}^{L}\mathbf{1}[C\models\varphi_{l}]. This measure \mu_{C,\varphi} captures the fraction of specification clauses met by the candidate C . 3.2 Composite Verification and Feedback Oracle To guide the refinement process, we rely on a verification oracle \mathcal{O} that provides multi-faceted feedback. This oracle leverages a test suite \{T_{j}\}_{j=1}^{M} , static analysis checks \{A_{k}\}_{k=1}^{N} , runtime logging mechanisms \mathcal{L}(C) , and formal verification conditions \mathcal{V}(C,\varphi) . Our ultimate goal is to minimize a composite error metric \delta(C,\varphi) that integrates these diverse signals. The composite metric ensures that improvement in one dimension (e.g., passing more tests) does not blind us to regressions in another (e.g., new structural violations). Concretely, we define: \delta(C,\varphi)=\alpha_{1}\epsilon_{\text{test}}(C,\varphi)+\alpha_{2}\epsilon_{\text{struct}}(C)+\alpha_{3}\epsilon_{\text{verify}}(C,\varphi)+\alpha_{4}\epsilon_{\text{logs}}(C), where \alpha_{i}>0 are weighting factors chosen to reflect the relative importance of tests, structural integrity, formal verification, and runtime logging anomalies. The testing error \epsilon_{\text{test}}(C,\varphi) accounts for weighted test failures: \epsilon_{\text{test}}(C,\varphi)=\frac{\sum_{j=1}^{M}\beta_{j}\mathbf{1}[\neg T_{j}(C)]}{\sum_{j=1}^{M}\beta_{j}}. Here, \beta_{j}>0 weights the importance of the j -th test, ensuring critical tests exert stronger influence on our optimization process. To incorporate architectural and syntactic considerations, we let: \epsilon_{\text{struct}}(C)=\frac{\sum_{k=1}^{N}w_{k}h_{k}(C)q_{k}}{\sum_{k=1}^{N}w_{k}}, where w_{k}>0 are structural check weights, h_{k}(C) severity measures, and q_{k} penalty factors for failing the k -th structural condition. Similarly, runtime anomalies integrate over execution time: \epsilon_{\text{logs}}(C)=\frac{\int_{0}^{T_{max}}\phi(\tau,C)\,d\tau}{T_{max}}, with \phi(\tau,C) quantifying logged anomalies at time \tau . Formal verification failures are aggregated as: \epsilon_{\text{verify}}(C,\varphi)=\frac{\sum_{m=1}^{M_{\varphi}}\zeta_{m}\mathbf{1}[\neg\mathcal{V}_{m}(C,\varphi)]}{\sum_{m=1}^{M_{\varphi}}\zeta_{m}}, where \zeta_{m} weights the significance of each verifiable property \mathcal{V}_{m} . Collectively, these definitions produce a highly granular view of correctness and demand satisfaction. They allow us to track progress across multiple dimensions simultaneously, ensuring that our refinement steps are globally informed rather than narrowly focused. 3.3 Contextual Constraints and Retrieval Mechanisms Since LLMs are bounded by a finite context window, the system employs a retrieval mechanism \Gamma(H_{t};Q)\to\Pi_{t} that selects a subset \Pi_{t} of historical information H_{t} relevant to the current query Q . This ensures: |\Pi_{t}|\leq\Lambda, where \Lambda is the maximum allowable context length. The selection is guided by a scoring