Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wild

Deepak Akkil, Mowafak Allaham, Amal Raj, Tamer Abuelsaad, Ravi Kokku · 2026

Read Emergence WebVoyager: Toward Consistent and Transparent Evaluation of (Web) Agents in The Wild by Deepak Akkil, Mowafak Allaham, Amal Raj, Tamer Abuelsaad, Ravi Kokku (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29020.txt.

Description

Akkil and co-authors introduce Emergence WebVoyager, a stricter version of the WebVoyager benchmark for web agents. Their audit of the original found vague task framing and variable operations. Those flaws hinder fair and reproducible comparisons. The new version standardizes task setup, failure handling, annotation, and reporting. Annotator agreement reached 95.9 percent. Applied to OpenAI Operator, the benchmark gave an overall success rate of 68.6 percent. That is well below the 87 percent reported before. Performance varied a lot across domains and task types.

Catalog record

CREATORDeepak Akkil, Mowafak Allaham, Amal Raj, Tamer Abuelsaad, Ravi Kokku
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39026
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.29020

Browse the full library · About the Tower of Babel