SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration
Jialong Chen 🖂, Xander Xu 🖂, Hu Wei 🖂, Chuan Chen 🖂, Bing Zhao 🖂 · 2026
Read SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration by Jialong Chen 🖂, Xander Xu 🖂, Hu Wei 🖂, Chuan Chen 🖂, Bing Zhao 🖂 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03823.txt.
Introduction
Jialong Chen and four colleagues grade coding agents on keeping a codebase alive, not just fixing it once. Real software matures through requirement changes and feature iterations over months, a reality one-shot repair benchmarks miss entirely. Their 2026 benchmark, SWE-CI, is the first built on the continuous integration loop: agents must shepherd real repositories through their actual development histories, dozens of rounds of analysis and coding per task.
One hundred tasks come from real projects whose histories average most of a year and dozens of consecutive commits. The guiding insight is that maintainability reveals itself over time, in how functional correctness holds up as the code evolves. An agent that patches brilliantly but poisons the future fails here. The benchmark offers the field its first systematic look at long-term code stewardship by machines.
Catalog record
| CREATOR | Jialong Chen 🖂, Xander Xu 🖂, Hu Wei 🖂, Chuan Chen 🖂, Bing Zhao 🖂 |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41602 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03823
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration Jialong Chen 🖂, Xander Xu 🖂, Hu Wei 🖂, Chuan Chen 🖂, Bing Zhao 🖂 Abstract Large language model (LLM)-powered agents have demonstrated strong capabilities in automating software engineering tasks such as static bug fixing. However, in the real world, the development of mature software is typically predicated on complex requirement changes and long-term feature iterations — a process that static, one-shot repair paradigms fail to capture. To bridge this gap, we propose SWE-CI , the first repository-level benchmark built upon the Continuous Integration loop, aiming to shift the evaluation paradigm for code generation from static, short-term functional correctness toward dynamic, long-term maintainability . The key insight is simple: Maintainability can be revealed by tracking how functional correctness changes over time. The benchmark comprises 100 tasks, each deriving from a real-world code repository with a development history spanning an average of 233 days and 71 consecutive commits. SWE-CI requires agents to systematically resolve these tasks through dozens of rounds of analysis and coding iterations. SWE-CI provides valuable insights into how well agents can sustain code quality throughout long-term evolution. 1 Introduction Automating software engineering has long been a central objective in the field of artificial intelligence. In recent years, breakthroughs in large language models (LLMs) have lent substantial momentum into this pursuit — from code completion and test generation to end-to-end program repair, LLM-driven agents have demonstrated capabilities rivaling those of human developers across multiple benchmarks. The concurrent evolution of coding benchmarks has played a pivotal role in this progress, providing both rigorous capability measurement and clear research direction. At the code generation level, HumanEval [ 6 ] , MBPP [ 2 ] and LiveCodeBench [ 11 ] established the single-file synthesis paradigm. At the repository level, SWE-bench [ 12 ] introduced the “Issue-to-PR” paradigm, requiring models to generate patches within complete repository contexts. At the agent interaction level, Terminal-bench [ 22 ] and \tau -bench [ 27 ] further broadened the evaluation scope to encompass terminal operations and multi-turn tool-use. Together, these efforts have established a multi-granularity and multi-scenario evaluation ecosystem for code intelligence. Despite the breadth and depth of this ecosystem, its underlying paradigm exhibits one fundamental limitation: Existing benchmarks almost exclusively focus on measuring the agent’s ability to write functionally correct code. However, in the real world, successful software is rarely achieved overnight; it is the result of long-term maintenance. Lehman’s Laws reveal that software quality inherently degrades as maintenance progresses [ 15 ] , while classic literature has established that maintenance activities account for 60% to 80% of total software lifecycle costs [ 4 ] . Therefore, there is an urgent need to design new benchmarks to effectively reflect the ability of models to maintain code. The reason this capacity has long been absent from evaluation is rooted in the prevailing benchmark paradigm itself. From HumanEval and LiveCodeBench to SWE-bench and Terminal-Bench, existing benchmarks universally adopt a snapshot-style protocol: the agent receives a single, complete requirement and produces a one-shot solution. Under this paradigm, an agent that hard-codes a brittle fix and one that writes clean, extensible code may both pass the same test suite — their difference in maintainability is simply invisible. It becomes visible only when the codebase must evolve: new requirements arrive, interfaces change, and modules must be extended. At that point, the cost of earlier design decisions compounds, and an agent that routinely produces poorly structured code will find each successive modification harder, eventually failing to keep pace. This yields a critical insight: an agent’s ability to maintain code can only be revealed through long-term evolution, where the consequences of past decisions accumulate over successive changes. Building upon this insight, we propose SWE-CI ( S oft W are E ngineering – C ontinuous I ntegration), a novel benchmark designed to evaluate how well agents maintain code across long-term code evolution. SWE-CI comprises 100 tasks, each defined by a base commit and a target commit from a real-world repository, with an average of 233 days and 71 consecutive commits of authentic evolutionary history in between. SWE-CI employs an Architect–Programmer dual-agent evaluation protocol: starting from the base commit, the agents execute a CI-loop that iteratively generates requirements, modifies source code, and runs tests, with the ultimate objective of passing all tests associated with the target commit. SWE-CI introduces EvoScore ( Evo lution Score ) as a proxy metric: it measures functional correctness on future modifications, so that agents whose earlier decisions facilitate subsequent evolution score higher, while those that accumulate technical debt see progressively declining performance. We conduct extensive experiments with a total consumption of more than 10 billion tokens. Results reveal that, despite substantial progress on functional correctness, state-of-the-art models still struggle when tasked with sustaining code quality over extended evolution. We further provide comprehensive, fine-grained analyses of the evaluation results, offering valuable insights into the coding capabilities of LLM-based agents and demonstrating the distinctive diagnostic value of SWE-CI. 2 Measuring the Agent’s Ability to Maintain Codebase 2.1 Task formalization We first establish a unified formalization for agentic coding tasks. Let t denote a single unit test and \mathcal{T}=\{t_{1},t_{2},\ldots,t_{|\mathcal{T}|}\} the set of all tests we are interested in. Let \mathcal{C} denote the space of codebases and \mathcal{R} the space of requirements. We further define two functions: • \mathsf{require}_{\mathcal{T}}:\mathcal{C}\times\mathcal{C}\to\mathcal{R} , which identifies the functional gap between two codebases with respect to \mathcal{T} , and produces a requirement document accordingly; • \mathsf{code}_{\mathcal{T}}:\mathcal{R}\times\mathcal{C}\to\mathcal{C} , which modifies a codebase according to a given requirement and returns an updated codebase. With these definitions, we observe that many prevailing coding benchmarks [ 6 , 2 , 11 , 12 , 22 , 27 ] follow the snapshot-based evaluation paradigm illustrated in Figure 1 . In this paradigm, the requirement depends only on the base codebase c_{0} and the “golden” codebase c_{\ast} , i.e., r\equiv\mathsf{require}_{\mathcal{T}}(c_{0},c_{\ast}) . However, in this paper, we turn to consider an evolution-based evaluation paradigm. The requirement in this paradigm is derived from the current codebase dynamically: r_{i}=\mathsf{require}_{\mathcal{T}}(c_{i},c_{*}) , and the codebase is updated accordingly: c_{i+1}=\mathsf{code}_{\mathcal{T}}(c_{i},r_{i}) . This iterative loop ensures that the consequences of earlier modifications propagate into subsequent iterations, making the agent’s long-term decision quality observable. Figure 1 : Unlike previous benchmarks, SWE-CI proposes an evolution-based evaluation. The red and blue arrows represent the actions of functions \mathsf{require} and \mathsf{code} , respectively. Dashed lines indicate processes that are unknown to the user. 2.2 Normalized Change Most benchmarks treat passing all test cases as the gold standard for functional correctness. However, in software engineering, some features require long-term planning and incremental development rather than a one-shot implementation. Moreover, during evolution it is common for previously passing tests to be inadvertently broken — a phenomenon known as regression . We therefore need a finer-grained metric that reflects the current state of a codebase c , rather than a binary pass/fail verdict. To this end, we introduce the normalized change . Let n(c) be the number of test cases that c passes: n(c)=\sum_{t\in\mathcal{T}}\mathbb{I}(t,c), (1) where the indicator function \mathbb{I}(t,c) equals 1 if and only if unit test t passes on codebase c , and 0 otherwise. The normalized change is then defined as: a(c)=\left\{\begin{aligned} \frac{n(c)-n(c_{0})}{n(c_{\ast})-n(c_{0})}&\,\,\,\,\text{if}\,\,n(c)\geq n(c_{0}),\\ \frac{n(c)-n(c_{0})}{n(c_{0})}&\,\,\,\,\text{else}.\end{aligned}\right. (2)