TerminalWorld : Benchmarking Agents on Real-World Terminal Tasks
Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou, Han Li, Chao Peng, Peter O’Hearn, Earl T. Barr, Mark Harman, Federica Sarro, He Ye · 2026
Read TerminalWorld : Benchmarking Agents on Real-World Terminal Tasks by Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou, Han Li, Chao Peng, Peter O’Hearn, Earl T. Barr, Mark Harman, Federica Sarro, He Ye (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-22535.txt.
Introduction
Chu and team launch TerminalWorld, a data engine making test tasks from in-the-wild shell logs. From 80,870 logs, the engine yields 1,530 checked tasks spanning 18 kinds, from short jobs to 50-step workflows, with 1,280 commands. A Checked subset of 200 tasks is picked by hand. Tests on TerminalWorld-Checked across eight models and six agents show systems still struggle with real shell workflows, reaching only 62.5%. TerminalWorld shows shell skills distinct from expert-picked benchmarks, with weak score link (Pearson r=0.20).
Catalog record
| CREATOR | Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou, Han Li, Chao Peng, Peter O’Hearn, Earl T. Barr, Mark Harman, Federica Sarro, He Ye |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 63935 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.22535
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TerminalWorld : Benchmarking Agents on Real-World Terminal Tasks Zhaoyang Chu, Jiarui Hu, Xingyu Jiang, Pengyu Zou, Han Li, Chao Peng, Peter O’Hearn, Earl T. Barr, Mark Harman, Federica Sarro, He Ye Abstract We introduce TerminalWorld , a scalable data engine that automatically reverse-engineers high-fidelity evaluation tasks from “in-the-wild” terminal recordings. Processing 80,870 terminal recordings, the engine yields a full benchmark of 1,530 validated tasks, spanning 18 real-world categories, ranging from short everyday operations to workflows exceeding 50 steps, and covering 1,280 unique commands. From these, we curate a Verified subset of 200 representative, manually reviewed tasks. Comprehensive benchmarking on TerminalWorld - Verified across eight frontier models and six agents reveals that current systems still struggle with authentic terminal workflows, achieving a maximum pass rate of only 62.5%. Moreover, TerminalWorld captures real-world terminal capabilities distinct from existing expert-curated benchmarks ( e.g. , Terminal-Bench), with only a weak correlation to their scores (Pearson r = 0.20 r=0.20 ). The automated engine makes TerminalWorld authentic and scalable by construction , enabling it to evaluate agents in real-world terminal environments as developer practices evolve. Data and code are available at https://github.com/EuniAI/TerminalWorld . 1 Introduction Terminal environments serve as a primary action space for autonomous agents to complete diverse tasks in complex software systems. Powered by advances in Large Language Models (LLMs) for multi-step reasoning and tool use, these agents are increasingly capable of automating terminal workflows by issuing commands, composing tools, and interpreting feedback within interactive CLI sessions, exemplified by open-source frameworks ( e.g. , SWE-agent ( Yang et al. 2024 ) , OpenHands ( Wang et al. 2025 ) ) and commercial CLI assistants ( e.g. , Claude Code ( Anthropic 2025 ) , Codex CLI ( OpenAI 2025 ) , Gemini CLI ( Google 2025 ) ). Yet, how to reliably evaluate these agents on real-world terminal tasks remains an open question. The prevailing answer has been manually curated benchmarks, such as Terminal-Bench ( Merrill et al. 2026 ) and LongCLI-Bench ( Feng et al. 2026 ) , where domain experts author tasks paired with executable environments. However, experts often tend to prioritize adversarial puzzles to artificially maximize difficulty, thereby diverging from authentic, real-world terminal workflows. Moreover, this labor-intensive process struggles to scale with evolving terminal practices and diverse emerging tools, leaving benchmarks narrowly scoped and quickly outdated. While recent automated synthesis methods ( Zhu et al. 2026 ; Lin et al. 2026 ; Gandhi et al. 2026 ; Pi et al. 2026 ; Wu et al. 2026 ) attempt to bypass this scalability bottleneck, they are primarily designed for training and rarely undergo the rigorous validation required to guarantee true authenticity. Therefore, we argue that an authentic, scalable evaluation system should be established to answer the key question: “How well do terminal agents perform on the real-world tasks that evolve alongside everyday practices?” To answer this question, we argue that naturally occurring terminal operations, if faithfully recorded, can be reverse-engineered into evaluation tasks that are authentic by construction. The asciinema platform 1 1 1 https://asciinema.org/ makes this feasible: developers voluntarily share terminal session recordings, each with a structured transcript capturing every command and its corresponding system response. These recordings form a self-curated, human-vetted, and continuously growing corpus of authentic developer workflows. To systematically exploit this resource, we introduce TerminalWorld , a data engine that operationalizes this insight: it automatically turns in-the-wild terminal recordings into executable, rigorously validated evaluation tasks. Turning raw recordings into evaluation tasks requires addressing three practical challenges, which our data engine systematically resolves: 1 Recordings are noisy and lack clear intent. Recording transcripts often contain typos, retries, and verbose system output, without explicit statements of the developer’s goal. We address this by distilling each transcript into two artifacts via an LLM ( e.g. , Claude Sonnet 4.6 ( Anthropic 2026a ) ): an outcome-oriented instruction that captures the developer’s underlying intent and a clean command script as the reference solution. 2 Recordings do not capture the underlying execution environments. The transcript captures commands but not the underlying system state of the author’s machine. We employ an LLM agent ( e.g. , Claude Code ( Anthropic 2025 ) ) to reverse-engineer this environment by inferring actual requirements while eliminating hallucinated dependencies. In particular, the agent physically builds a Docker image, launches the container, and replays the reference solution, using runtime failures as feedback for targeted repair. 3 Recordings lack an explicit test suite. While recordings naturally capture the execution trajectory, they lack an explicit test suite to automatically judge whether the task goal is achieved. Relying solely on LLMs to statically generate these tests is inherently vulnerable to false negatives (where correct solutions are rejected due to brittle tests) and false positives (where tasks can be trivially bypassed or solved by flawed workflows). We resolve this by equipping the agent with a trial-based refinement loop to generate and calibrate the test suite via actual execution feedback within the reproduced Docker container. Running this engine over 80,870 raw asciinema recordings yields 1,530 validated terminal tasks as the full TerminalWorld benchmark. The resulting tasks span 18 real-world terminal categories, range from short everyday operations to workflows exceeding 50 steps, and cover 1,280 unique commands, 91% of which are absent from Terminal-Bench. Since the pipeline is fully automated and asciinema keeps accumulating new uploads, TerminalWorld can be re-run as the platform grows, allowing it to scale with evolving developer practices. Unlike prior benchmarks, TerminalWorld is authentic and scalable by construction . From this collection, we curate a Verified 2 2 2 Here, “ Verified ” denotes manual review of label correctness, not proof of conformance to a formal specification. subset of 200 tasks, each cross-reviewed by the authors who manually execute the reference solution in the reproduced environment and audit the semantic alignment across all artifacts. While the full set of 1,530 tasks provides a representative snapshot of in-the-wild terminal usage, this Verified subset serves as a rigorous and challenging testbed for benchmarking frontier models and agents on complex, real-world terminal tasks. Through comprehensive benchmarking experiments using TerminalWorld - Verified across eight frontier LLMs ( e.g. , Claude Opus 4.7 ( Anthropic 2026b ) , GPT-5.5 ( OpenAI 2026 ) , Gemini 3.1 Pro ( Google 2026 ) ) and six leading terminal agents ( e.g. , Claude Code ( Anthropic 2025 ) , Codex CLI ( OpenAI 2025 ) , Gemini CLI ( Google 2025 ) ), our analysis reveals several key findings. 1 Frontier LLMs still struggle with real-world terminal tasks: even the best model solves only 62.5%, while failures expose an efficiency paradox , spending extra compute exploring authentic environments without making progress. 2 Agent frameworks mainly affect cost-effectiveness rather than the underlying capability ceiling, suggesting that practical agents for real-world terminal environments should reduce exploration friction rather than merely add orchestration complexity. 3 Terminal-Bench scores are only weakly predictive of agent performance on TerminalWorld - Verified (Pearson r=0.20 ), suggesting that existing expert-curated challenges do not fully capture the capabilities needed for real-world terminal workflows. 4 Although TerminalWorld tasks are grounded in real-world human recordings, agents often solve them through different valid command paths rather than mimicking the original human workflows, as reflected by an overall median command-set