DevOps-Gym : Benchmarking AI Agents in Software DevOps Cycle

Yuheng Tang, Kaijie Zhu, Bonan Ruan, Chuqi Zhang, Michael Yang, Hongwei Li, Suyue Guo, Tianneng Shi, Zekun Li, Christopher Kruegel, Giovanni Vigna, Dawn Song, William Yang Wang, Lun Wang, Yangruibo Ding, Zhenkai Liang, Wenbo Guo · 2026

Read DevOps-Gym : Benchmarking AI Agents in Software DevOps Cycle by Yuheng Tang, Kaijie Zhu, Bonan Ruan, Chuqi Zhang, Michael Yang, Hongwei Li, Suyue Guo, Tianneng Shi, Zekun Li, Christopher Kruegel, Giovanni Vigna, Dawn Song, William Yang Wang, Lun Wang, Yangruibo Ding, Zhenkai Liang, Wenbo Guo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20882.txt.

Introduction

Yuheng Tang and a large team test AI agents on the full life of software. Their 2026 paper presents DevOps-Gym, a benchmark for agents in the DevOps cycle. AI agents shine at writing code and fixing issues. Their skill at building, shipping, and running real software is still unknown. DevOps work demands more than code. It needs project-scale analysis and choices made in sequence.

An agent in this cycle has to work through large projects. It must grasp how programs behave as they run. It must use trade tools with care. And it must plan many steps in order. DevOps-Gym measures agents on that whole cycle. The test exposes gaps that pure coding tests hide. It gives labs a harder target for agent research.

Catalog record

CREATORYuheng Tang, Kaijie Zhu, Bonan Ruan, Chuqi Zhang, Michael Yang, Hongwei Li, Suyue Guo, Tianneng Shi, Zekun Li, Christopher Kruegel, Giovanni Vigna, Dawn Song, William Yang Wang, Lun Wang, Yangruibo Ding, Zhenkai Liang, Wenbo Guo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE66 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20882

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DevOps-Gym : Benchmarking AI Agents in Software DevOps Cycle Yuheng Tang, Kaijie Zhu, Bonan Ruan, Chuqi Zhang, Michael Yang, Hongwei Li, Suyue Guo, Tianneng Shi, Zekun Li, Christopher Kruegel, Giovanni Vigna, Dawn Song, William Yang Wang, Lun Wang, Yangruibo Ding, Zhenkai Liang, Wenbo Guo Abstract Even though demonstrating extraordinary capabilities in code generation and software issue resolving, AI agents’ capabilities in the full software DevOps cycle are still unknown. Different from pure code generation, handling the DevOps cycle in real-world software, including developing, deploying, and managing, requires analyzing large-scale projects, understanding dynamic program behaviors, leveraging domain-specific tools, and making sequential decisions. However, existing benchmarks focus on isolated problems and lack environments and tool interfaces for DevOps. We introduce DevOps-Gym , the first end-to-end benchmark for evaluating AI agents across core DevOps workflows: build and configuration, monitoring, issue resolving, and test generation, plus end-to-end pipeline tasks that require solving problems across all stages sequentially. DevOps-Gym includes 700 + 700+ real-world tasks collected from 30 + 30+ projects in Java and Go. We develop a semi-automated data collection mechanism with rigorous and non-trivial expert efforts in ensuring the task coverage and quality. Our evaluation of state-of-the-art models and agents reveals fundamental limitations: they struggle with issue resolving and test generation in Java and Go, and remain unable to handle new tasks such as monitoring and build and configuration. These results highlight the need for essential research in automating the full DevOps cycle with AI agents. 1 Introduction Software DevOps refers to the end-to-end process of developing, delivering, deploying, and managing software projects. It is a critical yet labor-intensive process. With recent advances in LLMs and AI agents, the development part can be largely automated (e.g., LLMs can automatically generate code, resolve GitHub issues) ( 46 ; 54 ; 48 ; 43 ; 22 ; 38 ) . However, the subsequent operational stages, such as system building, deployment, and monitoring, still demand substantial manual intervention. Automating these complex operational tasks presents a distinct set of challenges that transcend traditional code generation. It requires the ability to analyze system runtime behavior, interact with domain-specific tools, and execute multi-step plans. For instance, diagnosing a memory leak necessitates a sequence of actions: repeatedly invoking monitoring tools (e.g., using ps to inspect process state, iostat to identify I/O bottlenecks) to track memory resource usage and interpreting the output to identify signals of potential anomalies. Such tasks, which depend on complex tool use, dynamic interaction with a live environment, and coherent decision-making, make them a compelling application domain for autonomous AI agents. While numerous benchmarks exist for evaluating LLMs and AI agents, they predominantly focus on software development tasks such as code generation (e.g., HumanEval ( 4 ) ), issue resolving (e.g., SWE-bench ( 15 ) ), and test generation (e.g., SWT-bench ( 29 ) ). In contrast, work on the operational side remains limited and is often confined to simulated environments ( 14 ) or narrow infrastructure settings ( 5 ) . Consequently, there exists a significant gap for a benchmark capable of evaluating the end-to-end DevOps capabilities of agents in realistic software projects. Furthermore, few benchmarks are explicitly designed for agentic systems, which must execute multi-step workflows that integrate code reasoning and generation with tool usage. Existing benchmarks also lack dynamic execution environments with tool-calling interfaces that enable realistic agent interaction and evaluation. To address these gaps, we introduce DevOps-Gym , the first benchmark designed to evaluate AI agents across core DevOps workflows using real-world repositories and various DevOps tools. Unlike existing benchmarks that focus on single tasks in isolation with synthetic environment, DevOps-Gym provides: (1) Coverage of four essential DevOps stages that form the minimum viable pipeline: build and configuration for project deployment, monitoring for runtime problem detection, issue resolving for problem fixing, and test generation for patch validation; (2) Real-world tasks sourced from GitHub issues or synthesized tasks that mimic the complex real-world issue patterns; (3) Agentic workflows that require extensive tool use and multi-step planning; (4) A tool-augmented dynamic evaluation environment with standardized tool-calling interfaces and detailed metrics for different types of tasks. We make several design choices and efforts to address the key technical challenges of constructing DevOps-Gym . First, we manually analyze a large set of real-world issues, particularly for monitoring, build and configuration stage, to categorize representative issue types and summarize failure patterns for crafting synthetic tasks. Second, we apply a rigorous filtering process to prevent data contamination. Third, we invest extensive expert engineering effort to reproduce tasks to ensure their correctness, which required reconstructing environments, dependencies, configurations, and specific inputs. This was a multi-round, time-consuming process. Even with coding agent assistance, it often exceeds 10 hours of expert work per task, especially with incomplete reports. Finally, we carefully designed our evaluation metrics to enable rigorous and scalable evaluation, and provided standardized tool interfaces in the terminal-bench format ( 39 ) . With these extensive efforts, we craft 704 tasks collected from 30+ real-world projects in Java and Go. Additionally, we create 14 end-to-end pipeline tasks that chain all four stages sequentially, testing agents’ ability to maintain context across complete DevOps workflows. Our evaluation of three widely used coding agents across 5 LLMs and 4 agentic frameworks shows that even state-of-the-art systems fall short. The top-performing agent achieves success rates of just 51.85 % on build and configuration, 20.56 % on monitoring, 23.87 % on issue resolving, and 13.87 % on test generation. To the best of our knowledge, DevOps-Gym is the first end-to-end DevOps benchmark featuring agent-specific tasks alongside a comprehensive evaluation platform, including environments, tool interfaces, and metrics. We will open-source DevOps-Gym , together with its evaluation framework and baseline implementations, and will continue to improve it to facilitate future research on AI agents for broader software engineering. The key findings are shown below. • Agents frequently fail at high-level planning, struggling to formulate correct sequences of actions for multi-step building and monitoring workflows. • Agents consistently fail to use DevOps-specific tools correctly, particularly for building and monitoring. We hypothesize that these tools are out-of-distribution (OOD) for the base LLMs, which are seldom trained on relevant tool-use trajectories. • Agents exhibit a limited ability to parse and reason about dynamic information, such as program states, runtime logs, and system status. They also struggle with long-context reasoning required for complex tasks such as monitoring. • For issue resolving and test generation, although SOTA agents report strong performance on existing Python-based benchmarks, their performance drops significantly on our Java and Go tasks. We believe this may be due in part to data contamination. It also indicates that LLMs are not well-equipped to handle non-script languages (e.g., Java and Go), which involve compilation, more complex dependencies, and syntax. 2 Related Work Coding benchmarks. Function-level coding benchmarks ( 4 ; 28 ; 13 ; 27 ) are widely used as standard benchmarks for evaluating LLMs’ code generation capabilities. Some recent efforts move toward repository-level evaluation, which is more complex and relevant to real-world applications ( 51 ; 55 ; 6 ; 23 ; 25 ; 12 ; 20 ; 56 ; 52 ) . For example, RepoCod ( 26 ) assesses whether LLMs can handle multi-file code generation rather than isolated functions.

Read the complete text