ComputerRL : Scaling End-to-End Online Reinforcement Learning for Computer Use Agents
Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, Jie Tang · 2025
Read ComputerRL : Scaling End-to-End Online Reinforcement Learning for Computer Use Agents by Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, Jie Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14040.txt.
Introduction
Hanyu Lai and co-authors train computer-use agents that run desktop software on their own. Desktops are built for people, so agents must mix code calls with direct clicks and typing. Their framework, ComputerRL, joins both modes in one paradigm. Training at scale needs many desktops running at once without stalls or collapse.
Their setup runs thousands of virtual desktops in parallel for online training. A strategy that swaps learning rounds with fine-tuning rounds stops the policy from going stale. Applied to two open models, the method hit a new best score on a desktop task benchmark. The gain was clear for general agents, not just tuned ones. Code and a new office task set are shared.
Catalog record
| CREATOR | Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, Jie Tang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-14040
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ComputerRL : Scaling End-to-End Online Reinforcement Learning for Computer Use Agents Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, Jie Tang Abstract We introduce ComputerRL , a framework for autonomous desktop intelligence that enables agents to operate complex digital workspaces skillfully. ComputerRL features the API-GUI paradigm, which unifies programmatic API calls and direct GUI interaction to address the inherent mismatch between machine agents and human-centric desktop environments. Scaling end-to-end RL training is crucial for improvement and generalization across diverse desktop tasks, yet remains challenging due to environmental inefficiency and instability in extended training. To support scalable and robust training, we develop a distributed RL infrastructure capable of orchestrating thousands of parallel virtual desktop environments to accelerate large-scale online RL. Furthermore, we propose Entropulse, a training strategy that alternates reinforcement learning with supervised fine-tuning, effectively mitigating entropy collapse during extended training runs. We employ ComputerRL on open models GLM-4-9B-0414 and Qwen2.5-14B, and evaluate them on the OSWorld benchmark. The AutoGLM-OS -9B based on GLM-4-9B-0414 achieves a new state-of-the-art accuracy of 48.1% , demonstrating significant improvements for general agents in desktop automation. The algorithm and framework are adopted in building AutoGLM ( Liu et al. 2024a ) . 1 1 footnotetext: HL, XL and YZ contributed equally. 2 2 footnotetext: Work done while these authors interned at Zhipu AI. (a) The success rates of agents on OSWorld. (b) ComputerRL training reward curves (95% CIs). Figure 1: ComputerRL enables efficient end-to-end online policy optimization for OS agents. (a) On OSWorld ( Xie et al. 2024 ) , AutoGLM-OS , trained with ComputerRL , outperforms state-of-the-art agents. (b) Our Entropulse approach yields higher average training rewards and improves both learning efficiency and final performance over conventional methods. Figure 2: Examples of AutoGLM-OS ’s execution on four user tasks, including image processing between GIMP and LibreOffice Writer, monitoring system resource usage in Terminal, table calculation in LibreOffice Calc, and document formatting in LibreOffice Writer. 1 Introduction Large Language Models (LLMs) ( Achiam et al. 2023 ; Grattafiori et al. 2024 ; GLM et al. 2024 ; Zeng et al. 2022 ; Team et al. 2023 ; DeepSeek-AI et al. 2025 ; Bai et al. 2023a ; Yang et al. 2025 ) have dramatically expanded the scope and depth of artificial intelligence (AI) capabilities, driving a profound re-examination of our understanding of machine intelligence. Among all scenarios, the emergence of LLM-based GUI (graphical user interfaces) agents, capable of independently perceiving, reasoning, and executing complex tasks on user devices, has aroused particular interests from researchers ( Xi et al. 2023 ; Wang et al. 2023 ; Liu et al. 2023 ) . Given that desktops remain central to intelligence-intensive tasks, developing effective computer use agents is crucial for fundamentally transforming human-computer interactions ( Agashe et al. 2025 ; Wu et al. 2024 ) and lifting capabilities of AI to higher levels. Despite previous attempts to develop computer use agents ( Agashe et al. 2025 ; Lei et al. 2024 ; Xie et al. 2025 ) , enabling them to operate autonomously over extended periods in real-world scenarios remains a significant challenge. The first primary obstacle arises from the fact that GUIs are inherently designed for human interaction, making the simulation of human actions by GUI agents ( Liu et al. 2024b ; OpenAI 2025 ; Qin et al. 2025 ) a non-trivial and cumbersome endeavor. Second, current mainstream approaches of behavior cloning (BC) ( Bain & Sammut 1995 ; Bratko et al. 1995 ) , including manual annotation ( He et al. 2025 ) and model distillation ( Sun et al. 2024 ; Xu et al. 2024 ) , are limited in scalability and effectiveness. Manual annotation, while precise, is prohibitively labor-intensive for complex tasks. Model distillation, on the other hand, is constrained by the performance of the teacher models, limiting overall capability. Both methods typically exhibit poor generalization and limited error recovery abilities. Finally, although reinforcement learning (RL) has shown potential for desktop automation tasks ( Lu et al. 2025 ; Feng et al. 2025 ) , its practical application remains restricted due to computational complexity and methodological challenges. Complex environment, slow convergence, and known inefficiencies in RL training ( Xie et al. 2024 ; Bonatti et al. 2024 ; Yu et al. 2025 ; Fu et al. 2025 ) severely limit its large-scale adoption in training computer use agents. In this work, we propose ComputerRL , an end-to-end algorithmic framework designed to advance desktop-level planning, reasoning, and device operation, including a new API-GUI interaction paradigm, a scalable RL training infrastructure for computer environment, and a RL algorithm for extended effective training. First, we introduce API-GUI, a large-scale, automatically constructed API ecosystem that enables the agent to transcend the inherent biases of human-oriented operational paradigms. It instead leverages a more machine-oriented approach for device interaction, which combines API calls and GUI actions and thereby significantly enhances both the versatility and overall performance of the agent. Second, we develop a distributed training infrastructure utilizing virtual machine clusters based on Docker and gRPC protocols for scalability, which is fully compatible with AgentBench ( Liu et al. 2023 ) . This infrastructure supports thousands of parallel environments , ensuring high scalability and consistent interactions across all environments. Additionally, we combine the training infrastructure with the AgentRL framework ( Zhang et al. 2025 ) for efficient asynchronous training, substantially accelerating the training process. Finally, to counteract stagnation and convergence issues in extended RL training—specifically, entropy collapse and rising KL divergence—we propose Entropulse, which alternates between RL and periodic SFT phases. This approach maintains exploratory capacity and ensures continuous performance gains (Figure 1(b) ). As a result, by harnessing end-to-end RL and optimization in the desktop environment, ComputerRL has achieved remarkable improvement in understanding and operating GUIs. Evaluation on the OSWorld benchmark ( Xie et al. 2024 ) shows ComputerRL ’s significant improvements (see Figure 1(a) ) in computer use challenges, achieving a success rate of 48.1% (with 64% performance gain from RL), outperforming other state-of-the-art models including OpenAI CUA o3 (42.9%), UI-TARS-1.5 (42.5%), and Anthropic Claude Sonnet 4 (30.7%). In summary, our contributions are as follows: • We propose a new interaction paradigm, shift from human-centric to machine-oriented interaction frameworks by introducing a large-scale, automatically constructed API ecosystem integrated with conventional GUI operations. This approach addresses the inherent mismatch between human-designed interfaces and artificial agent capabilities, while achieving superior operational efficiency and generalization performance on computer-based tasks. • We establish a large-scale, distributed RL infrastructure for computer use agents by reconstructing virtual machine clusters, achieving unprecedented scalability with thousands of parallel environments and seamless AgentBench compatibility, thereby overcoming the critical bottleneck that has limited RL-based computer use agent training to scale experiments and enabling breakthrough results in large-scale agent training. • We introduce Entropulse, a novel training methodology that systematically addresses the challenges of entropy collapse and KL divergence accumulation in extended RL training through strategic alternation between RL and SFT phases, enabling sustained performance improvements and achieving state-of-the-art performance on the OSWorld benchmark. 2 The ComputerRL Framework The human-oriented design of GUI environments leads to inefficient interactions for agents, which impedes effective operation and learning. Furthermore, limited environment scalability