Surfer 2 The Next Generation of Cross-Platform Computer Use Agents
M. Andreux, M. Bakler, Y. Barbier, H. Benchekroun, E. Biré, A. Bonnet, R. Bordie, N. Bout, M. Brunel, A. Cambray, P.-L. Cedoz, A. Chassang, G. Cloix, E. Connelly, A. D. Constantinou, R. De Coster, H. de La Jonquière, A. Delfosse, M. Delpit, A. Deprez, A. Derupti, M. Diaz, S. D’Souza, J. Dujardin, A. · 2025
Read Surfer 2 The Next Generation of Cross-Platform Computer Use Agents by M. Andreux, M. Bakler, Y. Barbier, H. Benchekroun, E. Biré, A. Bonnet, R. Bordie, N. Bout, M. Brunel, A. Cambray, P.-L. Cedoz, A. Chassang, G. Cloix, E. Connelly, A. D. Constantinou, R. De Coster, H. de La Jonquière, A. Delfosse, M. Delpit, A. Deprez, A. Derupti, M. Diaz, S. D’Souza, J. Dujardin, A. (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19949.txt.
Introduction
Mathieu Andreux and fifty-two co-authors introduce Surfer 2, a computer-use agent that operates software purely by looking at the screen, across web, desktop, and mobile. Earlier agents leaned on platform-specific interfaces, blocking cross-platform use. Surfer 2 combines hierarchical context management, separate planning and execution, and self-verification with adaptive recovery to survive long tasks.
The system set state-of-the-art results on four benchmarks, all without task-specific fine-tuning. It scored about ninety-seven percent on WebVoyager and about eighty-seven percent on AndroidWorld. It reached about seventy percent on WebArena and about sixty percent on OSWorld. Given multiple attempts it exceeded human performance on each benchmark. The authors credit careful orchestration of foundation models, while calling for better vision-language models to cut costs.
Catalog record
| CREATOR | M. Andreux, M. Bakler, Y. Barbier, H. Benchekroun, E. Biré, A. Bonnet, R. Bordie, N. Bout, M. Brunel, A. Cambray, P.-L. Cedoz, A. Chassang, G. Cloix, E. Connelly, A. D. Constantinou, R. De Coster, H. de La Jonquière, A. Delfosse, M. Delpit, A. Deprez, A. Derupti, M. Diaz, S. D’Souza, J. Dujardin, A. |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19949
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Surfer 2 The Next Generation of Cross-Platform Computer Use Agents M. Andreux, M. Bakler, Y. Barbier, H. Benchekroun, E. Biré, A. Bonnet, R. Bordie, N. Bout, M. Brunel, A. Cambray, P.-L. Cedoz, A. Chassang, G. Cloix, E. Connelly, A. D. Constantinou, R. De Coster, H. de La Jonquière, A. Delfosse, M. Delpit, A. Deprez, A. Derupti, M. Diaz, S. D’Souza, J. Dujardin, A. Edmund, M. Eickenberg, A. Fatalot, W. Felissi, I. Herring, X. Koegler, E. Le Jumeau de Kergaradec, A. Lac, M. Langevin, C. Lauverjat, A. Loison, A. Manevich, A. Moyal, A. Nguyen Kerbel, M. Parovic, J. Revelle, G. Richard, M.L. Richter, R. Riochet, M. Santos, R. Savidan, L. Sifre, M. Theillard, M. Thibault, I. Valentini, T. Wu, L. Yie, K. Yuan, J. Zubovskij Abstract Building agents that generalize across web, desktop, and mobile environments remains an open challenge, as prior systems rely on environment-specific interfaces that limit cross-platform deployment. We introduce Surfer 2, a unified architecture operating purely from visual observations that achieves state-of-the-art performance across all three environments. Surfer 2 integrates hierarchical context management, decoupled planning and execution, and self-verification with adaptive recovery, enabling reliable operation over long task horizons. Our system achieves 97.1 97.1 % accuracy on WebVoyager, 69.6 69.6 % on WebArena, 60.1 60.1 % on OSWorld, and 87.1 87.1 % on AndroidWorld, outperforming all prior systems without task-specific fine-tuning. With multiple attempts, Surfer 2 exceeds human performance on all benchmarks. These results demonstrate that systematic orchestration amplifies foundation model capabilities and enables general-purpose computer control through visual interaction alone, while calling for a next-generation vision language model to achieve Pareto-optimal cost-efficiency. Surfer 2 The Next Generation of Cross-Platform Computer Use Agents M. Andreux, M. Bakler, Y. Barbier, H. Benchekroun, E. Biré, A. Bonnet, R. Bordie, N. Bout, M. Brunel, A. Cambray, P.-L. Cedoz, A. Chassang, G. Cloix, E. Connelly, A. D. Constantinou, R. De Coster, H. de La Jonquière, A. Delfosse, M. Delpit, A. Deprez, A. Derupti, M. Diaz, S. D’Souza, J. Dujardin, A. Edmund, M. Eickenberg, A. Fatalot, W. Felissi, I. Herring, X. Koegler, E. Le Jumeau de Kergaradec, A. Lac, M. Langevin, C. Lauverjat, A. Loison, A. Manevich, A. Moyal, A. Nguyen Kerbel, M. Parovic, J. Revelle, G. Richard, M.L. Richter, R. Riochet, M. Santos, R. Savidan, L. Sifre, M. Theillard, M. Thibault, I. Valentini, T. Wu, L. Yie, K. Yuan, J. Zubovskij H Company — Alphabetical order October 2025 Figure 1 : Surfer 2 state-of-the-art performance on WebVoyager, WebArena, OSWorld E2E, and AndroidWorld. Human performance is indicated when available. 1 Introduction Recent advances in Large Language Models (LLMs) and Vision-Language Models (VLMs) unlocked remarkable reasoning capabilities for agentic use cases [ 1 ] . However, turning these capabilities into reliable, general-purpose agents that operate autonomously in Graphical User Interfaces (GUIs) on complex, real-world tasks remains challenging. In recent years, one dominant path has been to train increasingly large models with minimal scaffolding like tool-call [ 2 , 3 ] to improve agentic capabilities [ 4 , 5 , 6 ] . In contrast, this work presents an alternative perspective: with proper orchestration and system design, existing state-of-the-art models can achieve human-level performance and exceed prior systems across multiple benchmarks. Prior approaches require environment-specific adaptations, such as DOM parsers for web navigation, accessibility trees for mobile interfaces, or specialized APIs for desktop applications, limiting generalization across diverse digital environments. This work introduces Surfer 2 , a unified, hierarchical agent architecture designed for complex tasks across desktop, web, and mobile environments using purely visual interaction. Surfer 2 comprises three components: Orchestrator (optional high-level planner), a Navigator (low-level GUI executor), and a Validator (evaluation module). Surfer 2 integrates third-party frontier models and H Company’s Holo1.5 models [ 7 ] in a design that separates long-term strategic planning from short-term tactical execution. A key design insight in Surfer 2 is architectural flexibility. It can enable or bypass the Orchestrator to match task complexity. For long-horizon problems, the Orchestrator runs as a high-level planner in the plan-and-act style [ 8 ] , where it decomposes the user task into verifiable goals, plans ahead, and delegates targeted subtasks to the Navigator. For simple tasks, the Orchestrator is bypassed and the Navigator is invoked directly. Built on our previous SurferH agent [ 9 ] , the Navigator follows a ReAct (reason+act) loop [ 10 ] . It perceives the environment purely via screenshots, reasons about the next step, and executes constrained keyboard and mouse-level controls with pixel-accurate UI localization from Holo1.5. Upon subtask completion, a Validator inspects the latest screenshots, the execution history, and the proposed answer to assess subtask success in one of two ways: (1) if the Orchestrator is enabled, it leverages the Validator’s report and either advances to the next subgoal or replans accordingly, or (2) if the Orchestrator is disabled, the Validator’s feedback is sent directly to the Navigator, which integrates it into its reasoning and continues the ReAct loop until the task is completed or a termination condition is reached. Without task-specific fine-tuning, Surfer 2 attains state-of-the-art results on four major benchmarks spanning different Computer Use environments (web browser, desktop, mobile): WebVoyager [ 11 ] and WebArena [ 12 ] , OSWorld [ 13 ] , and AndroidWorld [ 14 ] . On OSWorld and AndroidWorld, Surfer 2 surpasses the human baseline, underscoring that expert agent design is as crucial as model capability. 2 Related Work The development of Computer Use agents capable of controlling computers, web browsers, and mobile devices represents a key frontier in AI. In many real-world scenarios, agents encounter tools and software for which no API or Model Context Protocol (MCP) is available, leaving GUIs as the only viable control surface. Consequently, recent research has focused on enabling general-purpose agents to perceive, reason about, and act within GUIs, transforming visual interaction into a universal interface for autonomous computer use. 2.1 Agents for GUI Control The development of agents capable of controlling computers through their graphical interfaces has been a long-standing goal in AI. Early work focused on script-based or rule-based systems for automating specific, repetitive tasks [ 15 ] . Subsequent research introduced reinforcement learning (RL) and computer vision techniques for GUI automation [ 16 , 17 ] . Recent advances leverage large language models (LLMs) and vision-language models (VLMs) for more generalized and adaptable control [ 11 , 13 ] . 2.1.1 Browser Use Agents Web navigation agents are a highly active research area, with benchmarks like WebVoyager [ 11 ] and WebArena [ 12 ] providing standardized evaluation environments. Early methods often relied on interpreting the Document Object Model (DOM) to understand a page’s structure and content [ 18 , 19 , 12 , 20 ] . While effective, these text-based approaches struggle with visually-rich elements, dynamic content, or situations where visual layout and context are crucial for task success. Our work operates on a fundamentally different, multimodal principle: we use image-based states (screenshots) as the primary input, following the approach of Surfer-H [ 9 ] . This enables our agent to perceive and interact with the digital environment in a more human-like way, leveraging the visual understanding of large multimodal models (LMMs). Previous works have already explored this path; for instance Set-of-Marks [ 11 ] augments screenshots