RoadmapBench : Evaluating Long-Horizon Agentic Software Development Across Version Upgrades
Xinbo Xu, Ruihan Yang, Haiyang Shen, Wendong Xu, Bofei Gao, Ruoyu Wu, Kean Shi, Weichu Xie, Xuanzhong Chen, Ming Wu, Jason Zeng, Michael Heinrich, Elvis Zhang, Liang Chen, Kuan Li, Baobao Chang, [0.8em] UniPat AI Peking University Fudan University The University of Hong Kong Tsinghua University 0G Labs Pipeline Lab · 2026
Read RoadmapBench : Evaluating Long-Horizon Agentic Software Development Across Version Upgrades by Xinbo Xu, Ruihan Yang, Haiyang Shen, Wendong Xu, Bofei Gao, Ruoyu Wu, Kean Shi, Weichu Xie, Xuanzhong Chen, Ming Wu, Jason Zeng, Michael Heinrich, Elvis Zhang, Liang Chen, Kuan Li, Baobao Chang, [0.8em] UniPat AI Peking University Fudan University The University of Hong Kong Tsinghua University 0G Labs Pipeline Lab (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15846.txt.
Introduction
Xinbo Xu and fifteen colleagues introduce RoadmapBench, a benchmark for long-horizon agentic software development across version upgrades. It holds 115 coding tasks drawn from real open-source upgrades across 17 repositories and five programming languages. Each task asks the agent to build a target version's features from a source snapshot, with a median change of 3,700 lines across 51 files. Testing thirteen frontier models, the authors find even the strongest, Claude-Opus-4.7, solves only 39.1 percent of tasks, against 5.2 percent for the weakest. Long-horizon development remains largely unsolved.
Catalog record
| CREATOR | Xinbo Xu, Ruihan Yang, Haiyang Shen, Wendong Xu, Bofei Gao, Ruoyu Wu, Kean Shi, Weichu Xie, Xuanzhong Chen, Ming Wu, Jason Zeng, Michael Heinrich, Elvis Zhang, Liang Chen, Kuan Li, Baobao Chang, [0.8em] UniPat AI Peking University Fudan University The University of Hong Kong Tsinghua University 0G Labs Pipeline Lab |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70767 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15846
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
RoadmapBench : Evaluating Long-Horizon Agentic Software Development Across Version Upgrades Xinbo Xu, Ruihan Yang, Haiyang Shen, Wendong Xu, Bofei Gao, Ruoyu Wu, Kean Shi, Weichu Xie, Xuanzhong Chen, Ming Wu, Jason Zeng, Michael Heinrich, Elvis Zhang, Liang Chen, Kuan Li, Baobao Chang, [0.8em] UniPat AI Peking University Fudan University The University of Hong Kong Tsinghua University 0G Labs Pipeline Lab Abstract Coding agents are increasingly deployed in real software development, where a single version iteration requires months of coordinated work across many files. However, most existing benchmarks focus predominantly on single-issue bug fixes from Python repositories, with coarse pass/fail evaluation outcomes, and thus fail to capture long-horizon, multi-target development at real engineering scale. To address this gap, we present RoadmapBench , a benchmark of 115 long-horizon coding tasks grounded in real open-source version upgrades across 17 repositories and 5 programming languages. Each task places the agent on a source-version code snapshot and provides a multi-target roadmap instruction requiring it to implement the functionality introduced in the target version, with a median modification of 3,700 lines across 51 files. We conduct a systematic evaluation on thirteen frontier models and find that even the strongest, Claude-Opus-4.7, resolves only 39.1% of tasks, while the weakest achieves merely 5.2%, in stark contrast to existing bug-fix benchmarks, suggesting that long-horizon software development remains a largely unsolved problem. 2 2 footnotetext: Corresponding authors: [email protected] , [email protected] , [email protected] Figure 1: RoadmapBench Leaderboard. Resolved rate of top-performing models evaluated with OpenHands across 115 multi-target software evolution tasks spanning 5 languages and 17 repositories. Even the best-performing model resolves only 39.1% of tasks. 1 Introduction Table 1: Comparison with related coding benchmarks. Scope: task granularity. Subtask Score: target-level completion scoring. Solution: oracle patch size (LoC). Benchmark #Tasks Lang. Scope Subtask Score Solution SWE-bench Verified 23 500 Python Commit ✗ \sim 33 LOC SWE-bench Pro 8 1,865 Multi Commit ✗ \sim 107 LOC FeatureBench 36 200 Python Commit ✗ \sim 790 LOC TerminalBench 20 89 Multi Task ✗ \sim 280 LOC SWE-EVO 29 48 Python Version ✗ \sim 611 LOC NL2Repo 9 104 Python Repo ✗ \sim 3,000 LOC ROADMAPBENCH 115 Multi Version ✓ (avg. 5 targets) \sim 3,700 LOC The rapid progress of large language models (LLMs) ( 2 ; 24 ; 13 ) has enabled a new generation of coding agents that can plan, edit, execute, and validate software in interactive development environments ( 33 ; 34 ; 16 ; 30 ) . As these agents move beyond isolated code generation and bug fixing, the central challenge increasingly lies in sustained, multi-target software development. Evaluation is therefore shifting from short-horizon defect repair to long-horizon feature implementation. This raises a critical question: how to evaluate an agent on multi-target, human-scale development work spanning weeks or months? Existing benchmarks have not kept pace with this shift (Table 1 ). Most current benchmarks remain short-horizon: SWE-bench ( 17 ) and SWE-bench Pro ( 8 ) evaluate isolated software engineering problems, with oracle solutions of \sim 33 and \sim 107 lines respectively, one to two orders of magnitude below the scale of real engineering work. Long-horizon attempts remain scarce and collapse each task to a single binary outcome, overlooking the multi-target structure that real version upgrades naturally exhibit, where developers coordinate multiple substantial changes within a single release cycle. Beyond scope and granularity, existing benchmarks remain concentrated in a limited set of heavily reused Python repositories ( 19 ; 11 ) , compounding contamination risk as popular codebases become more likely to appear in pre-training corpora. To tackle these challenges, we propose RoadmapBench , a benchmark of 115 long-horizon coding tasks grounded in real open-source version upgrades. Each task starts from a repository snapshot pinned to an earlier release and requires the agent to implement the behaviors introduced in the next release, with a median oracle modification of approximately 3,700 lines across multiple files and modules. We convert each upgrade into a multi-target roadmap with a median of 5 subtasks, specifying what to implement, including API signatures, parameter semantics, default values, and exception behavior, while withholding implementation details. Each subtask is verified by its own test suite and contributes to a weighted overall score, so partial progress is captured as a continuous value rather than a binary outcome. To broaden coverage, we curate 17 repositories across 5 programming languages, spanning data processing, web frameworks, ORMs, serialization, GUI toolkits, and developer tooling, with no overlap with existing benchmarks. To ensure that failures reflect genuine capability gaps rather than benchmark artifacts, we combine static validation with attribution-driven rollout-based quality control to separate task-side defects from model-side limitations and iteratively repair confirmed task issues. We evaluate thirteen frontier models on RoadmapBench and observe that no model comes close to solving the benchmark. As shown in Figure 1 , even the strongest model, Claude-Opus-4.7, resolves only 39.1% of tasks, while the weakest achieves merely 5.2%. By comparison, these systems attain 80%+ scores on SWE-bench Verified ( 23 ) . The Completion Score reveals a consistent pattern: models routinely complete several subtasks before stalling at integration boundaries, offering cleaner separation across capability tiers than binary outcomes alone. In summary, our key contributions are as follows: • We construct RoadmapBench , a benchmark of 115 real open-source version-upgrade tasks across 17 repositories and 5 programming languages, establishing long-horizon multi-target software development as a distinct evaluation setting. • We develop a construction pipeline that transforms real version upgrades into multi-target tasks, and combines static validation with rollout-based quality control to separate task-side defects from genuine model limitations and iteratively repair confirmed issues. • We evaluate thirteen frontier models and find that resolved rates range from 5.2% to 39.1%, well below performance on existing bug-fix benchmarks, while Completion Score reveals fine-grained capability differences across domains and difficulty tiers beyond binary resolved metrics. 2 Related Work Coding Agents. LLM-based coding agents have evolved from single-turn code generation systems to interactive software engineering agents operating in realistic development environments ( 27 ; 10 ; 28 ) . OpenHands 31 provides an open platform for building generalist software development agents, while Terminus 2 20 serves as the reference agent implementation within the Harbor framework for autonomous evaluation in sandboxed environments. Commercial systems such as Claude Code 1 have further brought agentic coding into mainstream software development workflows. As these systems become increasingly capable, there is a growing need for benchmarks that better reflect the complexity of real-world software engineering. Coding Benchmarks for Agents. Coding benchmarks for LLM agents have progressively evolved from function-level synthesis to more realistic software engineering tasks. HumanEval 6 and MBPP 4 focus on function-level code generation. The SWE-bench family 17 ; 23 ; 8 extends evaluation to issue resolution and long-horizon engineering tasks in real-world repositories. Later benchmarks broaden evaluation to feature-oriented development and system-level interaction, including FeatureBench 35 and TerminalBench 20 . More recent work explores increasingly open-ended and long-horizon software engineering settings. NL2Repo 9 evaluates full repository generation from natural language specifications without requiring agents to evolve existing large-scale codebases, while SWE-EVO 29 studies Python version evolution but derives problem statements directly from release notes without explicit instruction-test alignment validation. Existing benchmarks still primarily evaluate isolated tasks rather than structured long-horizon multi-target software development processes. RoadmapBench