OpenComputer: Verifiable Software Worlds for Computer-Use Agents

Jinbiao Wei Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Qianran Ma P \hskip 1.00006pt{}^{{\color[rgb]{0.6016,0,0}\boldsymbol{P}}} Yilun Zhao Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Xiao Zhou Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Kangqi Ni C \hskip 1.00006pt{}^{{\color[rgb]{0.293,0.6133,0.8281}\boldsymbol{C}}} Guo Gan Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Arman Cohan Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}}, 𝑷 \hskip 1.00006pt{}^{{\color[rgb]{0.6016,0,0}\boldsymbol{P}}} University of Pennsylvania, 𝑪 \hskip 1.00006pt{}^{{\color[rgb]{0.293,0.6133,0.8281}\boldsymbol{C}}} University of North Carolina at Chapel Hill · 2026

Read OpenComputer: Verifiable Software Worlds for Computer-Use Agents by Jinbiao Wei Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Qianran Ma P \hskip 1.00006pt{}^{{\color[rgb]{0.6016,0,0}\boldsymbol{P}}} Yilun Zhao Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Xiao Zhou Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Kangqi Ni C \hskip 1.00006pt{}^{{\color[rgb]{0.293,0.6133,0.8281}\boldsymbol{C}}} Guo Gan Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Arman Cohan Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}}, 𝑷 \hskip 1.00006pt{}^{{\color[rgb]{0.6016,0,0}\boldsymbol{P}}} University of Pennsylvania, 𝑪 \hskip 1.00006pt{}^{{\color[rgb]{0.293,0.6133,0.8281}\boldsymbol{C}}} University of North Carolina at Chapel Hill (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19769.txt.

Introduction

Wei, Ma, Zhao, Zhou, Ni, Gan and Cohan build OpenComputer, a testbed for computer-use agents. It checks work against real software states instead of an AI judge. Four parts combine: state verifiers for apps, a self-improving verification layer, a pipeline that invents checkable desktop tasks, and a harness recording full runs with partial credit. Covering thirty-three apps and one thousand tasks, it matches human judges better than judge models. Leading agents still fail whole tasks, and open models fall sharply from their OSWorld scores.

Catalog record

CREATORJinbiao Wei Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Qianran Ma P \hskip 1.00006pt{}^{{\color[rgb]{0.6016,0,0}\boldsymbol{P}}} Yilun Zhao Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Xiao Zhou Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Kangqi Ni C \hskip 1.00006pt{}^{{\color[rgb]{0.293,0.6133,0.8281}\boldsymbol{C}}} Guo Gan Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Arman Cohan Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}}, 𝑷 \hskip 1.00006pt{}^{{\color[rgb]{0.6016,0,0}\boldsymbol{P}}} University of Pennsylvania, 𝑪 \hskip 1.00006pt{}^{{\color[rgb]{0.293,0.6133,0.8281}\boldsymbol{C}}} University of North Carolina at Chapel Hill
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53319
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.19769

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OpenComputer: Verifiable Software Worlds for Computer-Use Agents Jinbiao Wei Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Qianran Ma P \hskip 1.00006pt{}^{{\color[rgb]{0.6016,0,0}\boldsymbol{P}}} Yilun Zhao Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Xiao Zhou Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Kangqi Ni C \hskip 1.00006pt{}^{{\color[rgb]{0.293,0.6133,0.8281}\boldsymbol{C}}} Guo Gan Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}} Arman Cohan Y \hskip 1.00006pt{}^{{\color[rgb]{0,0.207,0.418}\boldsymbol{Y}}}, 𝑷 \hskip 1.00006pt{}^{{\color[rgb]{0.6016,0,0}\boldsymbol{P}}} University of Pennsylvania, 𝑪 \hskip 1.00006pt{}^{{\color[rgb]{0.293,0.6133,0.8281}\boldsymbol{C}}} University of North Carolina at Chapel Hill Abstract We present OpenComputer , a verifier-grounded framework for constructing verifiable software worlds for computer-use agents. OpenComputer integrates four components: (1) app-specific state verifiers that expose structured inspection endpoints over real applications, (2) a self-evolving verification layer that improves verifier reliability using execution-grounded feedback, (3) a task-generation pipeline that synthesizes realistic and machine-checkable desktop tasks, and (4) an evaluation harness that records full trajectories and computes auditable partial-credit rewards. In its current form, OpenComputer covers 33 desktop applications and 1,000 finalized tasks spanning browsers, office tools, creative software, development environments, file managers, and communication applications. Experiments show that OpenComputer’s hard-coded verifiers align more closely with human adjudication than LLM-as-judge evaluation, especially when success depends on fine-grained application state. Frontier agents struggle with end-to-end completion despite partial progress, and open-source models exhibit sharp drops from their OSWorld-Verified scores, exposing a persistent gap in robust computer automation. Figure 1: Overview of the OpenComputer verifiable software-world synthesis pipeline. Phase 1 generates app-specific verifier endpoints over the most reliable inspection channels and validates it with unit and integration tests for structured, machine-checkable state. Phase 2 closes a self-evolving loop: calibration tasks drive a strong agent run, an LLM evaluator and the programmatic verifier produce verdicts that disagreement analysis attributes, and verifier memory + checker/endpoint/doc fixes refine the verifier with execution-grounded feedback. Phase 3 proposes user goals, filters by complexity and data generatability, matches against the verifier, synthesizes the environment, and emits a final task instance. Phase 4 runs the agent and computes the reward. 1 Introduction Computer-use agents offer a promising path toward general-purpose AI systems that operate the same software interfaces humans use every day ( 1 ; 21 ; 2 ; 26 ) , but scaling their training and evaluation is limited by the cost of constructing realistic, reproducible desktop environments and tasks ( 34 ; 13 ) . Constructing a realistic desktop task involves far more than writing a natural-language instruction. A human developer must first design a plausible user goal, then manually prepare the underlying environment state ( e.g., creating or editing files, configuring folders, populating spreadsheets or documents, setting browser history or bookmarks, preparing emails or calendars), and ensures that the software state is both coherent and reproducible 32 ; 5 . These steps are tedious, application-specific, and difficult to standardize, making large-scale task creation slow and expensive. Beyond environment construction, computer-use tasks also require trustworthy verification of the resulting software state. In desktop settings, success is often reflected not only in visible screenshots, but also in application state, file contents, metadata, or persistent side effects 32 ; 5 . This makes evaluation difficult to scale: each task often requires custom inspection logic that can determine whether the intended state has actually been achieved. A natural fallback is to use an LLM-as-a-judge 19 ; 15 , but this introduces substantial limitations. LLM judgments can be sensitive to prompt wording, incomplete observations, and model-specific biases, and are often difficult to audit or reproduce across runs ( 30 ; 17 ; 29 ; 39 ) . More importantly, an LLM judge may reward outcomes that appear plausible from screenshots while missing errors in the underlying software state ( 28 ; 7 ) . Thus, scalable synthesis for computer-use agents must be coupled with reliable inspection rather than weak proxy evaluation. To address the dual bottlenecks of scalable environment construction and trustworthy state verification, we present OpenComputer , a verifier-grounded framework for synthesizing verifiable software worlds for computer-use agents. Rather than treating verification as a downstream evaluation detail, OpenComputer makes verification the organizing principle of environment and task construction. It consists of four tightly coupled components as illustrated in Figure 1 . First, it builds app-specific state verifiers that undergo a strict debug-fix-retry testing loop to reliably inspect software state through stable interfaces, defining exactly which task outcomes can be checked programmatically. Second, it further improves these verifiers through an execution-grounded self-evolution loop: calibration tasks are executed in sandboxed desktops, programmatic verifier outputs are compared against criterion-level LLM judgments, and verifier-side failures are used to refine checker logic, endpoints, or documentation. Third, on top of this verifier stack, OpenComputer synthesizes realistic user tasks through a structured pipeline that filters for difficulty, data generatability, and state inspectability. Finally, OpenComputer provides an evaluation harness that runs agents in fresh desktop sandboxes, records full screenshot-action trajectories, and scores each run by executing verifier commands over the resulting software state. Empirically, OpenComputer shows that current computer-use agents still struggle to reliably complete realistic desktop tasks end to end. GPT-5.4 achieves the strongest overall performance, with a full task success rate of 68.3%, while Claude-Sonnet-4.6 and Kimi-K2.6 reach 64.4% and 58.8%, respectively. Open-source agents lag substantially behind, with especially large drops relative to their reported performance on existing desktop benchmarks such as OSWorld 32 . Our analysis further highlights the importance of verifier-grounded benchmark construction. Hard-coded verifiers align more closely with human adjudication than an agentic LLM judge, particularly when success depends on fine-grained application state that cannot be reliably inferred from screenshots alone. We summarize our contributions as follows: 1. We introduce OpenComputer, a verifier-grounded framework for synthesizing realistic software worlds for computer-use agents, where the task descriptions, environments, and verifiers for evaluation are all automatically generated without relying on manual construction. 2. We empirically validate the reliability of this construction pipeline, showing that verifier-grounded evaluation aligns more closely with human adjudication than LLM-as-judge evaluation, and that the self-evolving verification layer can identify and repair verifier-side failures. 3. We instantiate a large-scale benchmark spanning 33 desktop applications and 1,000 finalized tasks, and evaluate frontier and open-source computer-use agents to show that realistic, verifier-grounded desktop workflows remain challenging for current systems. 2 Related Work Benchmarks for Computer-Use Agents. Prior benchmarks for computer-use agents fall into two main categories: static trajectory datasets and interactive task environments. Static datasets such as Mind2Web ( 9 ) and Android in the Wild ( 25 ) provide broad coverage of web or mobile interfaces through human demonstrations, but primarily evaluate offline action prediction. Interactive benchmarks more directly evaluate agents through environment feedback, including OSWorld ( 32 ) and Windows Agent Arena ( 5 ) for desktop operating-system tasks, BEARCUBS ( 27 ) , RealWebAssist ( 36 ) for web tasks, WebArena ( 40 ) and VisualWebArena ( 16 ) for realistic web navigation, WorkArena ( 10 ) and Scuba ( 8 ) for enterprise web workflows, and AndroidWorld ( 24 ) for mobile control. However, these benchmarks are still largely human-curated and often limited by the number of task instances, application domains, or manually written reward checks. In contrast, OpenComputer focuses on scaling computer-use environment construction itself. Synthetic Environments for Agents. Recent work increasingly treats environment construction as a key bottleneck for training interactive agents. In tool-use and function-calling settings, AgentScaler builds simulated, database-backed API environments ( 11 ) , Agent World Model scales

Read the complete text