LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command - Line Interfaces
Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Yu Qiao, Shenglin Zhang, Kaipeng Zhang, [2mm] NKU · 2026
Read LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command - Line Interfaces by Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Yu Qiao, Shenglin Zhang, Kaipeng Zhang, [2mm] NKU (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-14337.txt.
Description
Yukang Feng and 18 co-authors introduce a 2026 arXiv benchmark for long-horizon coding agents that work through command lines. Existing benchmarks suffer short task horizons, contamination from GitHub scraping, and coarse evaluation, failing to test the long-horizon planning real software engineering demands. LongCLI-Bench fills that gap.
Twenty high-quality, long-horizon tasks were curated from over 1,000 computer science assignments and real workflows, spanning from-scratch builds, feature addition, bug fixing, and refactoring. A dual-set protocol measures requirement fulfillment through fail-to-pass tests and regression avoidance through pass-to-pass tests, with step-level scoring to pinpoint failures. Even state-of-the-art agents scored below 20% pass rates, showing how far long-horizon agentic programming still has to go.
Catalog record
| CREATOR | Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Yu Qiao, Shenglin Zhang, Kaipeng Zhang, [2mm] NKU |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-14337