LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command - Line Interfaces

Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Yu Qiao, Shenglin Zhang, Kaipeng Zhang, [2mm] NKU · 2026

Read LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command - Line Interfaces by Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Yu Qiao, Shenglin Zhang, Kaipeng Zhang, [2mm] NKU (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-14337.txt.

Description

Yukang Feng and 18 co-authors introduce a 2026 arXiv benchmark for long-horizon coding agents that work through command lines. Existing benchmarks suffer short task horizons, contamination from GitHub scraping, and coarse evaluation, failing to test the long-horizon planning real software engineering demands. LongCLI-Bench fills that gap.

Twenty high-quality, long-horizon tasks were curated from over 1,000 computer science assignments and real workflows, spanning from-scratch builds, feature addition, bug fixing, and refactoring. A dual-set protocol measures requirement fulfillment through fail-to-pass tests and regression avoidance through pass-to-pass tests, with step-level scoring to pinpoint failures. Even state-of-the-art agents scored below 20% pass rates, showing how far long-horizon agentic programming still has to go.

Catalog record

CREATORYukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Yu Qiao, Shenglin Zhang, Kaipeng Zhang, [2mm] NKU
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-14337

Browse the full library · About the Tower of Babel