Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wild
Deepak Akkil, Mowafak Allaham, Amal Raj, Tamer Abuelsaad, Ravi Kokku · 2026
Read Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wild by Deepak Akkil, Mowafak Allaham, Amal Raj, Tamer Abuelsaad, Ravi Kokku (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29020.txt.
Description
Akkil and co-authors introduce Emergence WebVoyager, a stricter version of the WebVoyager benchmark for web agents. Their audit of the original found vague task framing and variable operations. Those flaws hinder fair and reproducible comparisons. The new version standardizes task setup, failure handling, annotation, and reporting. Annotator agreement reached 95.9 percent. Applied to OpenAI Operator, the benchmark gave an overall success rate of 68.6 percent. That is well below the 87 percent reported before. Performance varied a lot across domains and task types.
Catalog record
| CREATOR | Deepak Akkil, Mowafak Allaham, Amal Raj, Tamer Abuelsaad, Ravi Kokku |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39026 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.29020