SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering
Qingnan Ren, Shun Zou, Shiting Huang, Ziao Zhang, Kou Shi, Zhen Fang, Yiming Zhao, Yu Zeng, Qisheng Su, Lin Chen, Yong Wang, Zehui Chen, Xiangxiang Chu, Feng Zhao, [4pt] University of Science, Technology of China · 2026
Read SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering by Qingnan Ren, Shun Zou, Shiting Huang, Ziao Zhang, Kou Shi, Zhen Fang, Yiming Zhao, Yu Zeng, Qisheng Su, Lin Chen, Yong Wang, Zehui Chen, Xiangxiang Chu, Feng Zhao, [4pt] University of Science, Technology of China (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17526.txt.
Introduction
Ren and colleagues introduce SaaSBench, the first benchmark probing coding agents on long-horizon enterprise SaaS engineering. It spans 30 tasks, six domains, eight languages, six databases and thirteen frameworks. A dependency-aware hybrid evaluation gives fine-grained repeatable scores of multi-component systems. Experiments reveal the bottleneck is not code logic but configuring and integrating whole systems. Over 95 percent of failures strike before deep business logic, through early halts during setup or fruitless debug loops.
Catalog record
| CREATOR | Qingnan Ren, Shun Zou, Shiting Huang, Ziao Zhang, Kou Shi, Zhen Fang, Yiming Zhao, Yu Zeng, Qisheng Su, Lin Chen, Yong Wang, Zehui Chen, Xiangxiang Chu, Feng Zhao, [4pt] University of Science, Technology of China |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 98896 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.17526
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SaaSBench: Exploring the Boundaries of Coding Agents in Long-Horizon Enterprise SaaS Engineering Qingnan Ren, Shun Zou, Shiting Huang, Ziao Zhang, Kou Shi, Zhen Fang, Yiming Zhao, Yu Zeng, Qisheng Su, Lin Chen, Yong Wang, Zehui Chen, Xiangxiang Chu, Feng Zhao, [4pt] University of Science, Technology of China Abstract As autonomous coding agents become capable of handling increasingly long-horizon tasks, they have gradually demonstrated the potential to complete end-to-end software development. Although existing benchmarks have recently evolved from localized code editing to from-scratch project generation, they remain confined to structurally simplified, single-stack applications. Consequently, they fail to capture the heterogeneous environments, full-stack orchestration, and system-level complexity of real enterprise Software as a Service (SaaS) systems, leaving a critical gap in assessing agents under realistic engineering constraints. To fill this gap, we introduce SaaSBench, the first benchmark designed to explore the boundaries of AI agents in enterprise SaaS engineering. Spanning 30 complex tasks across 6 SaaS domains with 5,370 validation nodes, it incorporates 8 programming languages, 6 databases, and 13 frameworks to meticulously mirror real-world software heterogeneity. Furthermore, we design a dependency-aware hybrid evaluation paradigm tailored for complex systems with long horizons and multi-component coupling, enabling fine-grained, reproducible assessment. Crucially, our extensive experiments reveal a striking insight: the primary bottleneck for state-of-the-art agents is not generating isolated code logic, but successfully configuring and integrating a multi-component system. Over 95% of task failures occur before agents even reach deep business logic, with models often falling victim to overconfidence and prematurely halting during foundational system setup, or getting trapped in ineffective debugging loops. We hope SaaSBench serves as a practical and challenging testbed to drive the evolution of reliable, system-level coding agents. The code is available at https://github.com/ShadeCloak/SaaSbench . 1 Introduction With the rapid development of large language models (LLMs) [ 2 , 4 , 40 , 20 , 27 ] , coding agents have evolved from early tools primarily designed for function completion and localized editing into systems with composite capabilities, including requirement understanding, system design, code generation, environment interaction, and iterative debugging [ 11 , 27 , 3 , 1 , 41 ] . They are also entering real software development workflows in diverse forms [ 13 , 26 , 45 , 7 , 48 , 19 ] . At the same time, coding agents continue to lower the technical barriers to software development, enabling users without development experience to drive the construction of complete software systems from scratch through natural language requirements [ 14 , 42 , 43 ] . Figure 1 : Up-to-Date Leaderboard: Coding agent performance on SaaSBench evaluation tasks. Meanwhile, corresponding benchmarks continue to evolve. As shown in Table 1 , this trajectory aligns with the expanding capabilities of coding agents. Existing code benchmarks can be broadly divided into two categories. The first mainly focuses on localized and isolated software engineering tasks, such as function-level code generation, patch fixing, and localized modifications within repositories [ 6 , 5 , 17 , 25 , 30 , 21 , 9 , 28 ] . These benchmarks are better suited for measuring short-horizon and localized engineering behaviors, but they struggle to reflect the holistic capabilities required for end-to-end software development. The second category begins to examine the ability of agents to build complete code repositories or projects from scratch based on natural language requirements, thereby placing higher demands on long-horizon planning, cross-file coordination, and system-level consistency [ 23 , 29 , 10 , 39 , 31 , 12 , 32 ] . Although recent project-level and repository-level benchmarks have made progress, they still face three key limitations: 1. Lack of real-market grounding. Existing benchmarks typically define task instances first and then abstract categories from them. As a result, tasks often lack clear market origins, stable product categories, and well-defined business boundaries. This makes it difficult to assess whether an agent truly possesses the ability to build real commercial Software as a Service (SaaS) products. 2. Limited system complexity. Most existing benchmarks operate in software development settings centered on a single language, a single component, or weakly coupled architectures. In contrast, real SaaS system development typically requires the joint design and implementation of the frontend, backend, database, authentication, deployment, and cross-component workflows. 3. Insufficient evaluation mechanisms. Existing evaluations for end-to-end development tasks usually rely on flat end-to-end signals, such as execution outcomes and unit test pass rates. These evaluation methods lack clear definitions and sufficient constraints. They are suitable only for relatively simple software development tasks and fail to characterize prerequisite dependencies, state dependencies, and other constraints in complex real-world business workflows. To address these limitations, we introduce SaaSBench , the first coding agent benchmark systematically designed for real enterprise-level SaaS development scenarios. SaaSBench starts from real software development markets and their open-source product implementations, and constructs the benchmark through a rigorous multi-stage process with strict quality validation. It contains 30 task instances across 6 high-level SaaS domains, covering mainstream SaaS software development scenarios. Each task consists of a long-context product requirements document (PRD), an ambiguity-resolution knowledge base (KB), a standardized runtime environment, and an accompanying DAG-based test suite. This design evaluates whether coding agents can complete the full engineering loop from scratch, including requirement understanding, system implementation, debugging, deployment, and execution. Overall, the PRDs in SaaSBench contain approximately 4,363 lines on average. The benchmark includes 5,370 executable validation nodes and covers 8 programming languages, 6 database types, and 13 frontend and backend development frameworks, reflecting the complexity and diversity of real-world software development. Table 1 : Comparison of SaaSBench with representative coding agent benchmarks . ✓ fully incorporated; ✓ partially incorporated; ✗ not incorporated; – not applicable. Benchmark Task Realism System Complexity Evaluation Fidelity From Scratch Market- Grounded Deployable Runtime Multi- Language Language Cross- Component Workflow Depth PRD Lines Semantic Judge Workflow Dependency Snippet-Level Coding Benchmarks HumanEval [ 6 ] ✗ ✗ ✗ ✗ Python ✗ ✗ – ✗ ✗ MBPP [ 5 ] ✗ ✗ ✗ ✗ Python ✗ ✗ – ✗ ✗ APPS [ 17 ] ✗ ✗ ✗ ✗ Python ✗ ✗ – ✗ ✗ DS-1000 [ 22 ] ✗ ✗ ✗ ✗ Python ✗ ✗ – ✗ ✗ CodeContests [ 25 ] ✗ ✗ ✗ ✓ Py / Ja / C ++ ✗ ✗ – ✗ ✗ EvoCodeBench [ 24 ] ✗ ✗ ✗ ✗ Python ✗ ✗ – ✗ ✗ SWE-Bench [ 21 ] ✗ ✗ ✗ ✗ Python ✗ ✗ – ✗ ✗ GitTaskBench [ 36 ] ✗ ✓ ✗ ✗ Python ✗ ✓ – ✗ ✗ Repository- & Project-Level Coding Benchmarks ProjectEval [ 29 ] ✓ ✗ ✗ ✗ Python ✗ ✗ 34.45 ✗ ✗ NL2Repo-Bench [ 10 ] ✓ ✗ ✗ ✗ Python ✗ ✗ 2452.64 ✗ ✗ RepoGenesis [ 39 ] ✓ ✗ ✓ ✓ Py / Ja ✓ ✗ 210.17 ✗ ✗ PRDBench [ 12 ] ✓ ✗ ✗ ✗ Python ✗ ✗ 105.22 ✓ ✗ ProjDevBench [ 31 ] ✓ ✗ ✗ ✗ C ++ ✗ ✗ 283.85 ✓ ✗ SaaSBench (Ours) ✓ ✓ ✓ ✓ Py / Ja / Go … (8 langs.) ✓ ✓ 4362.7 ✓ ✓ In addition, we design a dependency-aware hybrid evaluation paradigm for long-horizon and highly interactive end-to-end system development tasks. The paradigm centers