Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks

Tajamul Ashraf, Amal Saqib, Hanan Gani, Muhra AlMahri, Yuhao Li, Noor Ahsan, Umair Nawaz, Jean Lahoud, Hisham Cholakkal, Mubarak Shah, Philip Torr, Fahad Shahbaz Khan, Rao Muhammad Anwer, Salman Khan · 2025

Read Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks by Tajamul Ashraf, Amal Saqib, Hanan Gani, Muhra AlMahri, Yuhao Li, Noor Ahsan, Umair Nawaz, Jean Lahoud, Hisham Cholakkal, Mubarak Shah, Philip Torr, Fahad Shahbaz Khan, Rao Muhammad Anwer, Salman Khan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-24876.txt.

Description

Ashraf and co-authors present Agent-X, a 2025 benchmark for testing AI agents on hard visual tasks that take many steps. Older benchmarks lean on made-up, single-turn questions and judge only the final answer. Agent-X holds 828 tasks with real images, image sets, videos, and written instructions. The tasks span six settings, among them web browsing, driving, sports, security, and math.

A scoring framework grades each step of the agent's work, not just the outcome. It checks whether each step was right, whether the chain of steps made sense, and whether tools were used well. Even the best current model families solved fewer than half of the tasks end to end. The authors point to weak multi-step reasoning and tool use as the main bottlenecks.

Catalog record

CREATORTajamul Ashraf, Amal Saqib, Hanan Gani, Muhra AlMahri, Yuhao Li, Noor Ahsan, Umair Nawaz, Jean Lahoud, Hisham Cholakkal, Mubarak Shah, Philip Torr, Fahad Shahbaz Khan, Rao Muhammad Anwer, Salman Khan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE127 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-24876

Browse the full library · About the Tower of Babel