MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers

Chaithanya Bandi, Ben Hertzberg, Geobio Boo, Tejas Polakam, Jeff Da, Sami Hassaan, Manasi Sharma, Andrew Park, Ernesto Hernandez, Dan Rambado, Ivan Salazar, Rafael Cruz, Chetan Rane, Ben Levin, Brad Kenstler, Bing Liu · 2026

Read MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers by Chaithanya Bandi, Ben Hertzberg, Geobio Boo, Tejas Polakam, Jeff Da, Sami Hassaan, Manasi Sharma, Andrew Park, Ernesto Hernandez, Dan Rambado, Ivan Salazar, Rafael Cruz, Chetan Rane, Ben Levin, Brad Kenstler, Bing Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-00933.txt.

Description

Chaithanya Bandi and colleagues introduce MCP-Atlas, a benchmark for tool-use competency against production MCP servers. Existing evaluations miss realistic multi-step workflows, breadth across real servers rather than mocks, and structured claim-level scoring free of verbosity or style. MCP-Atlas holds 1,000 tasks by human experts spanning 36 real servers and 220 tools, with prompts specifying no servers, tools, or parameters. Agents must find relevant tools among plausible distractors and compose multi-step, cross-server workflows, scored against atomic claims from tool outputs. Answer-centric scoring credits valid alternative trajectories.

Catalog record

CREATORChaithanya Bandi, Ben Hertzberg, Geobio Boo, Tejas Polakam, Jeff Da, Sami Hassaan, Manasi Sharma, Andrew Park, Ernesto Hernandez, Dan Rambado, Ivan Salazar, Rafael Cruz, Chetan Rane, Ben Levin, Brad Kenstler, Bing Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61798
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602.00933

Browse the full library · About the Tower of Babel