AgentSearchBench: A Benchmark for AI Agent Search in the Wild
Bin Wu, Arastun Mammadli, Xiaoyu Zhang, Emine YilmazCentre for Artificial Intelligence, University College London · 2026
Read AgentSearchBench: A Benchmark for AI Agent Search in the Wild by Bin Wu, Arastun Mammadli, Xiaoyu Zhang, Emine YilmazCentre for Artificial Intelligence, University College London (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22436.txt.
Introduction
Wu and three colleagues offer AgentSearchBench, a test for finding fitting AI agents in the wild. Agent skills are mixed and execution-dependent, so text alone judges them poorly. Past tests assume set functions, controlled pools or executable queries only. Built from nearly 10,000 real agents, it frames search as retrieval and ranking under executable queries and high-level descriptions, scored by execution. Tests show a gap between semantic similarity and actual performance, exposing limits of description-based methods. Lightweight behavioural signals, including execution-aware probing, improve ranking a lot.
Catalog record
| CREATOR | Bin Wu, Arastun Mammadli, Xiaoyu Zhang, Emine YilmazCentre for Artificial Intelligence, University College London |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47648 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.22436
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AgentSearchBench: A Benchmark for AI Agent Search in the Wild Bin Wu, Arastun Mammadli, Xiaoyu Zhang, Emine YilmazCentre for Artificial Intelligence, University College London Abstract The rapid growth of AI agent ecosystems is transforming how complex tasks are delegated and executed, creating a new challenge of identifying suitable agents for a given task. Unlike traditional tools, agent capabilities are often compositional and execution-dependent, making them difficult to assess from textual descriptions alone. However, existing research and benchmarks typically assume well-specified functionalities, controlled candidate pools, or only executable task queries, leaving realistic agent search scenarios insufficiently studied. We introduce AgentSearchBench, a large-scale benchmark for agent search in the wild, built from nearly 10,000 real-world agents across multiple providers. The benchmark formalizes agent search as retrieval and reranking problems under both executable task queries and high-level task descriptions, and evaluates relevance using execution-grounded performance signals. Experiments reveal a consistent gap between semantic similarity and actual agent performance, exposing the limitations of description-based retrieval and reranking methods. We further show that lightweight behavioral signals, including execution-aware probing, can substantially improve ranking quality, highlighting the importance of incorporating execution signals into agent discovery. Our code is available at https://github.com/Bingo-W/AgentSearchBench . 1 Introduction The rapid emergence of AI agentic systems is reshaping how humans accomplish complex tasks by enabling execution to be delegated to autonomous agents across a wide range of domains ( 7 ; 12 ) . Modern agents can reason, plan, and interact with external tools and services to complete multi-step objectives ( 15 ; 9 ; 23 ) . This progress has led to a rapidly expanding ecosystem of agentic components, ranging from general-purpose assistants to highly specialized task-oriented modules. As humans increasingly rely on agents developed by diverse third-party providers, a fundamental challenge arises: how can suitable agents be reliably identified and selected for a given task? Addressing this challenge is critical not only for end users seeking effective task completion, but also for developers and orchestration systems aiming to compose scalable and robust agentic workflows ( 11 ; 14 ) . However, identifying suitable agents is inherently challenging. Compared to traditional tools whose functionality is typically scoped to specific operations ( 32 ) , agent capabilities are often more compositional and execution-dependent, making them difficult to assess without observing task outcomes. As a result, textual descriptions provide only a partial signal of real competence ( 26 ; 38 ; 8 ) : agents with similar descriptions may perform differently in practice, while semantically dissimilar agents can achieve comparable results. This semantic–performance misalignment is further amplified in large and open agent ecosystems, where overlapping functionalities and non-uniform description formats make capability comparison difficult ( 42 ) . Consequently, agent search is fundamentally more complex than conventional tool retrieval or model selection. Despite growing interest in agentic systems, existing research and benchmarks have not yet provided a realistic setting for studying agent search (in Table 1 ). Prior work on tool retrieval and related benchmarks primarily assumes that functionality can be inferred from structured descriptions or well-specified interfaces ( 24 ; 32 ) , which does not capture the compositional and execution-dependent nature of agent capabilities. Meanwhile, recent studies on automated agentic system design typically evaluate methods in small-scale or controlled environments where candidate agents are clearly differentiated ( 30 ; 41 ) . Such assumptions differ substantially from open ecosystems, where many agents exhibit overlapping capabilities and must be selected under uncertainty. Furthermore, existing tool and agent selection benchmarks largely focus on executable task queries with predefined inputs and outputs, whereas real-world agent discovery often begins from high-level task descriptions that are not directly executable. As a result, performance-grounded agent search in realistic open ecosystems remains insufficiently studied. To bridge these gaps, we introduce AgentSearchBench, a large-scale benchmark for agent search built from nearly 10,000 real-world agents. AgentSearchBench captures the diversity of open agent ecosystems by including agents from different providers with varying description styles, capability granularity, and functional overlap. Built on this resource, we formalize agent search as retrieval and reranking problems under both executable task queries and high-level task descriptions. Crucially, agent relevance is defined using execution-grounded performance signals rather than textual similarity. We further develop a scalable evaluation pipeline that generates task instances and converts execution outcomes into fine-grained relevance annotations for both retrieval and ranking assessment. Through extensive benchmarking experiments, we reveal a consistent gap between semantic similarity and actual task performance, providing empirical evidence for the execution-dependent nature of agent capabilities. Retrieval and reranking methods that rely primarily on matching task descriptions with agent documentation often fail to surface high-performing agents, particularly when search begins from high-level task descriptions where capability requirements are implicit. To better understand this limitation, we further study execution-aware probing, which augments description-based ranking with lightweight behavioral signals obtained from agent execution. Results show that even limited probing can substantially improve ranking quality, highlighting the importance of incorporating execution signals into realistic agent discovery pipelines. Our contributions can be summarized as follows: (1) We formulate agent search as a new retrieval and ranking problem under execution-dependent capability uncertainty. (2) We construct AgentSearchBench, a large-scale benchmark with nearly 10,000 real-world agents, supporting both executable task queries and high-level task descriptions under an execution-grounded evaluation framework. (3) We provide extensive empirical analysis revealing a substantial semantic–performance gap and demonstrate the effectiveness of lightweight behavioral probing for improving agent ranking. Agent/LLM/Tool #Candidates Realistic Task Type ToolBench ( 24 ) Tool 16,464 ✓ Exec. ToolRet ( 32 ) Tool 43,215 ✓ Exec. TREC 2025 ( 17 ) LLM 1,131 ✓ Exec. AgentSquare ( 30 ) Agent 16 ✗ Exec. OKC Bench ( 41 ) Agent 127 ✗ Exec. AgentSearchBench Agent 9,759 ✓ Exec. / Non-exec. Table 1: Comparison of AgentSearchBench with tool and agent retrieval benchmarks. “Realistic” indicates whether agents/tools are sourced from real-world platforms. “Task Type” indicates support for executable (Exec.) or non-executable (Non-exec.) task specifications. 2 Related Work Agentic Systems and Orchestration. Recent advances in agentic systems have enabled autonomous agents to solve complex tasks through reasoning, planning, and tool interaction ( 15 ; 9 ; 23 ) . As agent ecosystems rapidly expand, a key challenge is how to select suitable agents for a given task. Existing work focuses on agent design and orchestration, including multi-agent collaboration ( 11 ; 14 ) and workflow composition frameworks ( 43 ; 30 ; 41 ) , typically assuming a predefined and limited set of candidates. However, in open ecosystems with many overlapping agents ( 42 ; 38 ; 44 ) , agent selection must be performed under significant capability uncertainty, motivating agent search as a distinct problem. Tool Retrieval and Selection. Existing works on tool retrieval and selection ( 24 ; 32 ) aim to identify suitable tools for task execution. These methods typically retrieve tools based on textual descriptions or structured schemas ( 34 ; 19 ; 25 ) , and are primarily evaluated on executable task queries with predefined inputs and outputs ( 24 ; 25 ) . While effective in such settings, these assumptions do not capture real-world agent search, which often begins from high-level and non-executable task descriptions. Moreover, agent capabilities are compositional, inconsistently documented, and execution-dependent, making textual similarity insufficient