Live-SWE-agent : Can Software Engineering Agents Self-Evolve on the Fly?

Chunqiu Steven Xia Zhe Wang Yan Yang Yuxiang Wei Lingming Zhang[] University of Illinois Urbana-Champaign, [] {chunqiu2, zhe36, ywei40, lingming}@illinois.edu [email protected] · 2025

Read Live-SWE-agent : Can Software Engineering Agents Self-Evolve on the Fly? by Chunqiu Steven Xia Zhe Wang Yan Yang Yuxiang Wei Lingming Zhang[] University of Illinois Urbana-Champaign, [] {chunqiu2, zhe36, ywei40, lingming}@illinois.edu [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-13646.txt.

Introduction

Chunqiu Steven Xia and 4 co-authors introduce Live-SWE-agent, a software agent that rewrites its own scaffold while it works on real coding tasks. Language model agents for software work normally ship with a fixed, hand-designed set of tools and prompts. Searching that design space by hand is slow and costly, and offline self-improvement may not transfer across models.

Live-SWE-agent starts from a minimal bash-only scaffold and evolves its own implementation on the fly, problem by problem. On the SWE-bench Verified benchmark it reached the best solve rate reported among software agents in the study, without test-time scaling. It also led the newer SWE-Bench Pro benchmark in their evaluation.

Catalog record

CREATORChunqiu Steven Xia Zhe Wang Yan Yang Yuxiang Wei Lingming Zhang[] University of Illinois Urbana-Champaign, [] {chunqiu2, zhe36, ywei40, lingming}@illinois.edu [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-13646

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Live-SWE-agent : Can Software Engineering Agents Self-Evolve on the Fly? Chunqiu Steven Xia Zhe Wang Yan Yang Yuxiang Wei Lingming Zhang[] University of Illinois Urbana-Champaign, [] {chunqiu2, zhe36, ywei40, lingming}@illinois.edu [email protected] Abstract Large Language Models (LLMs) are reshaping almost all industries, including software engineering. In recent years, a number of LLM agents have been proposed to solve real-world software problems. Such software agents are typically equipped with a suite of coding tools and can autonomously decide the next actions to form complete trajectories to solve end-to-end software tasks. While promising, they typically require dedicated design and may still be suboptimal, since it can be extremely challenging and costly to exhaust the entire agent scaffold design space. Recognizing that software agents are inherently software themselves that can be further refined/modified, researchers have proposed a number of self-improving software agents recently, including the Darwin-Gödel Machine (DGM). Meanwhile, such self-improving agents require costly offline training on specific benchmarks and may not generalize well across different LLMs or benchmarks. In this paper, we propose Live-SWE-agent , the first live software agent that can autonomously and continuously evolve itself on-the-fly during runtime when solving real-world software problems. More specifically, Live-SWE-agent starts with the most basic agent scaffold with only access to bash tools (e.g., mini-SWE-agent), and autonomously evolves its own scaffold implementation while solving real-world software problems. Our evaluation on the widely studied SWE-bench Verified benchmark shows that Live-SWE-agent can achieve an impressive solve rate of 77.4% without test-time scaling, outperforming all existing software agents, including the best proprietary solution. Moreover, Live-SWE-agent outperforms state-of-the-art manually crafted software agents on the recent SWE-Bench Pro benchmark, achieving the best-known solve rate of 45.8%. More details at: https://github.com/OpenAutoCoder/live-swe-agent 0 0 footnotetext: † Work done as a research intern at the University of Illinois Urbana-Champaign. * Customized SWE-agent ( 2 ) ** JSON-based apply_patch tool plus dynamic thinking time ( 1 ) Figure 1 : SWE-bench Verified and SWE-Bench Pro results (single attempt w/o test-time scaling) 1 Introduction Large Language Models (LLMs) have rapidly progressed from simple code autocompletion 7 ; 12 ; 22 ; 39 to interactive agents capable of navigating repositories, running tests, and submitting patches end - to - end 43 ; 35 ; 48 ; 38 ; 10 ; 23 . Early conversational repair systems leveraged environment feedback to iteratively refine candidate fixes 41 , while subsequent agentic frameworks (such as SWE-agent 43 and OpenHands 35 ) augmented LLMs with terminals, editors, and search, enabling multi - step tool use on complex repositories. At the same time, complementary “more agentless” pipelines (e.g., Moatless 27 and Agentless 38 ) argue that much of the perceived complexity in agent scaffold design can be replaced by dedicated prompting and workflow design. Despite this progress, most existing agents have fixed designs and are limited in their static action space, where the scaffold implementation (including tools) is preset even when a task could benefit from more customization. In addition, manually designing an optimal software agent scaffold is extremely challenging and costly due to the infinite design space. As a result, more recently, the community has begun to explore self-evolving software agents 46 ; 31 ; 34 , which iteratively modify their own scaffold implementations and empirically validate each change using offline evaluation signals from coding benchmarks. However, these approaches add significant additional cost. For example, a single run of DGM on SWE-bench costs around $22,000 according to the original paper 46 . Furthermore, they rely heavily on offline evolution, where improvements are typically learned on certain benchmarks and then baked into a static agent. In this way, the learned agents may become specialized to the given benchmarks and underlying LLMs, and may not generalize well beyond that. To bridge this gap, we introduce Live-SWE-agent , the first live , runtime self-evolving software engineering agent that expands and revises its own capabilities on the fly while working on a real-world issue. Our key insight is that software agents are themselves software systems, and modern LLM-based software agents already possess the intrinsic capability to extend or modify their own implementation at runtime. While our idea of enabling on-the-fly self-evolution applies to all parts of the agent scaffold implementation, as a first step, we focus primarily on tool creation, as it is one of the most essential parts of a software agent. Live-SWE-agent starts from a simplistic agent with only bash tool access (e.g., mini-SWE-agent 43 ). During the regular issue-solving loop, the agent can synthesize, modify, and execute custom tools such as editors, code search utilities, and domain-specific analyzers. A lightweight step - reflection prompt repeatedly asks the agent whether creating or revising a tool would accelerate progress, thereby turning tooling into a first - class decision alongside ordinary actions like running tests. This mechanism leaves the underlying scaffold unchanged, requires no offline training, and is agnostic to the underlying LLM. By elevating tool creation to an explicit, iterative decision point, we unlock this hidden on-the-fly self-improvement ability. Live-SWE-agent addresses the limitations of existing research. First, with tool creation, the agent action space adapts to the problem at hand, producing task-relevant tools that precisely capture the needs to accomplish the current task. Furthermore, by shifting improvement from offline training to online evolution, Live-SWE-agent alleviates tedious scaffold engineering because new abilities emerge from the encountered issues themselves. Importantly, tool synthesis is not a one-shot preprocessing action but a continuous iterative process interleaved with problem solving. The agent can refine tools as their understanding of the failure mode evolves. Despite its minimal and simplistic design, Live-SWE-agent is generalizable to different agent scaffolds and LLMs, with state-of-the-art open results on software issue solving. We have evaluated Live-SWE-agent on the widely used SWE-bench Verified benchmark 26 and the more challenging SWE-Bench Pro 4 . Without any test - time scaling, Live-SWE-agent achieves 77.4% resolve rate on Verified and 45.8% on Pro, surpassing state-of-the-art open - source baselines and even the best commercial agent systems. Our ablations and tooling analyses reveal that (1) custom tool creation materially improves effectiveness (higher solve rates) with minimal overhead, (2) benefits persist across different state-of-the-art LLM backends, with better results on stronger models, demonstrating its promising future as LLMs’ capabilities rapidly evolve, and (3) synthesized tools include both generic utilities and task-aligned specializations that benefit issue-specific problem-solving. In summary, we make the following contributions: • The first live software agent. We present Live-SWE-agent , the first live software agent that can autonomously self-evolve its own scaffold implementation on the fly when solving real-world issues, without any offline training or extra pipelines. • Minimal and general implementation. Our current implementation adopts a minimal and general design, where the agent starts from a minimal bash-only scaffold (e.g., mini-SWE-agent) and self-improves on the fly by creating general or customized tools. The design is compatible with any existing software agent loop or LLM with negligible overhead, has been publicly available at: https://github.com/OpenAutoCoder/live-swe-agent . • State-of-the-art performance. On SWE-bench Verified and SWE-Bench Pro, Live-SWE-agent reaches 77.4% and 45.8% solve rates (without any test-time scaling), respectively, outperforming all existing open-source agents and commercial systems at the time of writing. To our knowledge, our SWE-Bench Pro result is also the best reported to date. • Comprehensive analysis. We

Read the complete text