A Systematic Study of LLM-Based Architectures for Automated Patching DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering CCS: Software and its engineering Software development techniques CCS: Software and its engineering Automatic programming

Qingxiao Xu, Ze Sheng, Zhicheng Chen, Jeff Huang · 2026

Read A Systematic Study of LLM-Based Architectures for Automated Patching DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering CCS: Software and its engineering Software development techniques CCS: Software and its engineering Automatic programming by Qingxiao Xu, Ze Sheng, Zhicheng Chen, Jeff Huang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01257.txt.

Introduction

Xu and three colleagues compare four LLM-based patching designs in one controlled test: fixed workflow, single-agent, multi-agent, and general-purpose code agents. Judging patch correctness, failure modes, token use, and run time on real vulnerability tasks, they find clear trade-offs. Fixed workflows run lean but snap, single-agent systems mix flexibility with low cost, and multi-agent designs generalize better at higher overhead and drift risk on complex tasks. Unexpectedly, general-purpose code agents patch best overall, adapting across vulnerability types through general tool interfaces.

Catalog record

CREATORQingxiao Xu, Ze Sheng, Zhicheng Chen, Jeff Huang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67386
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.01257

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Systematic Study of LLM-Based Architectures for Automated Patching DOI: XXXXXXX.XXXXXXX Conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Software and its engineering CCS: Software and its engineering Software development techniques CCS: Software and its engineering Automatic programming Qingxiao Xu, Ze Sheng, Zhicheng Chen, Jeff Huang Abstract . Large language models (LLMs) have shown promise for automated patching of software security vulnerabilities, but their effectiveness depends strongly on how they are integrated into patching systems. While prior work explores prompting strategies and individual agent designs, the field lacks a systematic comparison of patching architectures. In this paper, we present a controlled evaluation of four LLM-based patching paradigms, including fixed workflow, single-agent system, multi-agent system, and general-purpose code agents, using a unified benchmark and evaluation framework. We analyze patch correctness, tool usage, token usage, and execution time across real-world vulnerability tasks. Surprisingly, general-purpose code agents achieve the strongest overall patching performance , benefiting from general-purpose tool interfaces that support effective adaptation across vulnerability types. Moreover, our results reveal clear architectural trade-offs: fixed workflows are efficient but brittle, single-agent systems balance flexibility and cost, and multi-agent designs improve generalization at the expense of substantially higher overhead and increased risk of reasoning drift on complex tasks. Overall, we show that architectural design and iteration depth, rather than model capability alone, dominate the reliability and cost of LLM-based automated patching. 1. Introduction Modern software ecosystems amplify the impact of vulnerabilities: a single flaw such as Log4Shell (CVE-2021-44228) in Apache Log4j or the recent MongoBleed vulnerability (CVE-2025-14847) in MongoDB, can cascade through dependency chains and compromise large numbers of downstream systems ( 7 ; 8 ) . While vulnerability detection is increasingly automated ( 41 ; 12 ; 28 ) , repair remains the dominant bottleneck : patches must preserve program semantics, eliminate exploitability, and integrate cleanly into large, evolving codebases. Fully automating this process is difficult because it requires deep semantic reasoning, precise fault localization, and rigorous validation, making scalable and correct vulnerability remediation an open challenge. Recent advances in large language models (LLMs) have begun to break this bottleneck. In DARPA’s AI Cyber Challenge (AIxCC) ( 9 ) , finalist teams demonstrated fully automated Cyber Reasoning Systems (CRSs) for real-world open-source projects that not only detected 77% (54/70) of injected vulnerabilities, but also correctly patched 61% (43/70) of them. Subsequent systems such as DeepMind’s CodeMender ( 25 ) and OpenAI’s Aardvark ( 23 ) further showed that LLM-driven pipelines (combined with fuzzing, symbolic execution, and formal verification) can generate human-quality, upstreamable patches. Together, these results mark a fundamental shift: vulnerability repair is no longer purely manual , but increasingly driven by autonomous, reasoning-centric AI systems. Despite this progress, critical questions remain unanswered. Although all AIxCC finalist CRSs leverage LLMs to generate patches, there is little understanding of how different architectural integration strategies affect repair effectiveness, efficiency, and robustness. Existing work ( 40 ; 33 ; 37 ; 18 ; 38 ) overwhelmingly treats system architecture as an implementation detail rather than a first-class design variable. Moreover, leading production systems (i.e., CodeMender and Aardvark) are proprietary, obscuring critical architectural decisions and making it difficult to assess whether their design choices generalize across vulnerability classes and codebases. In this paper, we address this gap by conducting the first architecture-centric study of LLM-based vulnerability patching systems. We identify and analyze four distinct architectural paradigms: three domain-specific patching architectures including fixed-workflow pipelines, single-agent systems and multi-agent systems, and general-purpose coding agents, which collectively capture the core architectural patterns used in the patching components of the seven AIxCC finalist CRSs; some systems adopt a single pattern, while others combine multiple architectures. To enable controlled comparison, we reimplement the first three architectures within a unified patching framework 1 1 1 Our code has been released as open source on https://anonymous.4open.science/r/Patch-Architectures-07C6 . and evaluate them on a collection of 19 delta-scan Java vulnerabilities from the AIxCC benchmark, including six large, complex, real-world Java projects (i.e., Apache Commons Compress, ZooKeeper, Log4j, Tika, PDFBox, and POI). We evaluate the fourth paradigm using Claude Code ( 3 ) as a representative general coding agent. Our results reveal clear trade-offs across different architectures for automated vulnerability repair. Surprisingly, general coding agents achieve the strongest overall patching performance and robustness, particularly on large codebases, but at significantly higher token cost. Claude Code successfully repairs 16/19 vulnerabilities, while the best patch-specific agent repairs at most 13/19 (multi-agent on GPT-5). Patch-specific agent architectures are also substantially more token-efficient and faster, yet more prone to brittle search behavior and vulnerability types as reasoning scope expands. Notably, multi-agent designs do not consistently outperform well-designed single-agent systems, indicating that coordination overhead can offset the benefits of agent specialization in vulnerability repair. These findings show that repair effectiveness depends not only on model capability, but critically on how reasoning, control, and tool usage are architected. In sum, our contributions are: • Architecture-level taxonomy. We identify and formalize four architectural paradigms for LLM-based vulnerability patching, and present their design and implementation in detail in Section 3. To the best of our knowledge, this is the first work to clearly describe, reimplement, and systematically evaluate these architectures under a unified experimental setting. • Unified evaluation on the AIxCC dataset. We develop a unified evaluation framework and compare representative systems across all four architectures on the AIxCC benchmark. Our results show that general coding agents achieve the best overall performance, while other architectures remain important by successfully patching complementary cases. • System behavior and cost analysis. We conduct an in-depth log-based analysis of system behavior, comparing tool usage, agent interactions, cost, and latency. We find that patch-specific agents are more token-efficient, while general coding agents show stronger robustness against infinite loops and long-running failures. 2. Background and Problem Setting 2.1. Automated Vulnerability Repair Automated Program Repair (APR) ( 17 ) aims to synthesize source-level patches that fix general software defects without introducing regressions. In the context of security, APR focuses on vulnerability repair : eliminating exploitable behaviors while preserving intended program semantics. Given the high cost and scalability limits of manual remediation, vulnerability repair has become an important problem in secure software maintenance. A substantial body of prior work addresses APR using both program analysis-driven techniques and LLM–based approaches ( 13 ) . We review representative systems and techniques in Section 5 . In a typical vulnerability repair setting, the system is provided with (i) a target program, (ii) evidence of a vulnerability, such as a crashing input, exploit trigger, or static analysis report, and (iii) optional validation mechanisms, including tests or runtime checks. The objective is to generate a patch that removes the vulnerability, passes validation, and integrates cleanly into the existing codebase. Unlike functional bug repair, security patching must reason about exploitability, adversarial inputs, and subtle control- and data-flow properties. As a result, correctness, robustness, and resistance to incomplete fixes are central concerns. 2.2. The AIxCC Patching Task DARPA has continuously organized competitions that emphasize end-to-end automation, with the goal of minimizing human intervention in the defense process. In the 2013–2016 Cyber Grand Challenge (CGC) ( 10 ) , representative systems such as Mayhem ( 4 ) and

Read the complete text