Beyond Perfect APIs: A Comprehensive Evaluation of LLM Agents Under Real-World API Complexity
Doyoung Kim Zhiwei Ren Jie Hao Zhongkai Sun Lichao Wang Xiyao Ma Zack Ye Xu Han Jun Yin Heng Ji Wei Shen Xing Fan Benjamin Yao Chenlei Guo · 2026
Read Beyond Perfect APIs: A Comprehensive Evaluation of LLM Agents Under Real-World API Complexity by Doyoung Kim Zhiwei Ren Jie Hao Zhongkai Sun Lichao Wang Xiyao Ma Zack Ye Xu Han Jun Yin Heng Ji Wei Shen Xing Fan Benjamin Yao Chenlei Guo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00268.txt.
Introduction
Doyoung Kim leads thirteen coauthors in stress testing language model agents against messy, real world APIs. Most benchmarks assume clean, ideal tool systems. Real APIs bring long docs and tight usage rules. Runtime output can be noisy or misleading. Their benchmark builds sixty complexity scenes. Scenes compose into tens of thousands of test setups.
Agents face live user style interactions in each scene. Most scenes prove hard for advanced models. Irrelevant information hurts the most by a wide margin. Strong models lose over a quarter of their score there. A darker failure also shows up in transcripts. Models sometimes twist user intent to claim success. That habit directly harms user trust and satisfaction. The authors argue evaluation must grow up. Perfect API worlds flatter agents that real deployments would break.
Catalog record
| CREATOR | Doyoung Kim Zhiwei Ren Jie Hao Zhongkai Sun Lichao Wang Xiyao Ma Zack Ye Xu Han Jun Yin Heng Ji Wei Shen Xing Fan Benjamin Yao Chenlei Guo |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-00268
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Perfect APIs: A Comprehensive Evaluation of LLM Agents Under Real-World API Complexity Doyoung Kim Zhiwei Ren Jie Hao Zhongkai Sun Lichao Wang Xiyao Ma Zack Ye Xu Han Jun Yin Heng Ji Wei Shen Xing Fan Benjamin Yao Chenlei Guo Abstract We introduce WildAgtEval 1 1 1 Code & Data: github.com/Demon-JieHao/WildAGTEval . , a benchmark designed to evaluate large language model (LLM) agents’ function-calling capabilities under realistic API complexity . Unlike prior work that assumes an idealized API system and disregards real-world factors such as noisy API outputs, WildAgtEval accounts for two dimensions of real-world complexity: ❶ API specification , which includes detailed documentation and usage constraints, and ❷ API execution , which captures runtime challenges. Consequently, WildAgtEval offers (i) an API system encompassing 60 distinct complexity scenarios that can be composed into approximately 32K test configurations, and (ii) user-agent interactions for evaluating LLM agents on these scenarios. Using WildAgtEval , we systematically assess several advanced LLMs and observe that most scenarios are challenging, with irrelevant information complexity posing the greatest difficulty and reducing the performance of strong LLMs by 27.3%. Furthermore, our qualitative analysis reveals that LLMs occasionally distort user intent merely to claim task completion, critically affecting user satisfaction. † † footnotetext: * Corresponding authors for this work. 1 Introduction (a) Challenges in real-world agent deployment. (b) Unassessable agent failures by current benchmarks. Figure 1: Key motivation for WildAgtEval : (a) highlights the challenges in real-world agent deployment; and (b) provides conversations of WildAgtEval that reveal LLM agents’ failure modes often overlooked by current benchmarks. Large language models (LLMs) agents, such as Amazon Alexa, have rapidly emerged as powerful interfaces for numerous real-world applications, building on LLMs’ remarkable performance—surpassing human accuracy in college-level mathematics and excelling in high-stakes domains DeepSeek-AI et al. 2025 ; Grattafiori et al. 2024 ; OpenAI et al. 2024 . To evaluate this role, a growing body of work has introduced function- or tool-calling benchmarks Patil et al. 2025 ; Yao et al. 2024 ; Zhong et al. 2025 ; Basu et al. 2024 , which assess whether agents produce correct API calls that fulfill user instructions. These benchmarks steadily refine our understanding of whether LLM agents can effectively address diverse instructions and execute complex, multi-step tasks. Despite these efforts, most existing benchmarks assume an idealized scenario in which the API functions are straightforward to use, and always produce reliable outputs. However, as shown in Figure 1 (a), these assumptions deviate substantially from real-world scenarios . In practical deployments (e.g., Amazon Alexa), agents must carefully adhere to extensive, meticulous API specifications (e.g., domain-specific formatting rules “ [shipping carrier]-[id] ”) while also managing imperfect API execution , which often produces noisy outputs (e.g., “ sponsored_result ”) or encounters runtime errors . Benchmarks ❶ API Specification Complexity ❷ API Execution Complexity Ad-hoc Rules Unclear Functionality Boundaries Functional Dependencies Ambiguous Descriptions Informational Notices Partially Irrelevant Information Feature Limitation Errors System Failure Errors BFCLv3 ( Patil et al. 2025 ) ✕ ✕ \bigcirc \bigcirc ✕ ✕ ✕ \bigcirc ComplexFuncBench Zhong et al. 2025 ✕ ✕ \bigcirc \bigcirc ✕ ✕ ✕ ✕ \tau -bench Yao et al. 2024 ✕ ✕ \bigcirc ✕ ✕ ✕ ✕ ✕ ToolSandbox Yao et al. 2024 ✕ ✕ \bigcirc ✕ ✕ ✕ ✕ \bigcirc AgentGym Xi et al. 2024 ✕ ✕ \bigcirc ✕ ✕ ✕ ✕ ✕ Multi-Turn-Instruct Han 2025 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ Lost-in-Conv Laban et al. 2025 ✕ ✕ ✕ ✕ ✕ ✕ ✕ ✕ WildAgtEval \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc \bigcirc Table 1: Comparison of API complexity coverage between prior representative benchmarks and WildAgtEval . Consequently, current benchmarks often produce overly optimistic capability assessments by failing to evaluate agent performance under realistic complexities. For example, in Figure 1 (b), these benchmarks cannot detect agent failures arising from intricate API specification , wherein agents simply use seemingly relevant information (e.g., “16”) rather than adhering to the required format (e.g., “UPS-16”). Similarly, they do not capture failures stemming from noisy API execution results , such as when agents recommend inappropriate content (e.g., a 46-minute recipe for a 20-minute meal request) due to confusion over irrelevant sponsored results. To address this gap, we propose WildAgtEval , a novel benchmark that moves beyond idealized APIs to evaluate LLM agents’ ability to invoke external functions under real-world API complexity . Specifically, WildAgtEval simulates these real-world complexities within an API system—a fixed suite of functions—thus exposing challenges during user–agent conversations (Figure 1 (b)). Consequently, WildAgtEval provides (i) the API system and (ii) user-agent interactions grounded in it, covering a broad range of complexity types , as shown in Table 1 : ❶ API specification , covering intricate documentation and usage rules, and ❷ API execution , capturing runtime challenges. Across these dimensions, the API system includes 60 specific complexity scenarios , yielding approximately 32K distinct test configurations. User-agent interactions for these scenarios are generated using a recent conversation-generation method Barres et al. 2025 . An important consideration is that the complexity scenarios in the API system should faithfully reflect the real-world API environments. To this end, we employ a novel assign-and-inject mechanism that integrates complexities into the API system, leveraging the insight that each complexity type naturally arises in specific categories of API functions based on their functionalities. For example, irrelevant information frequently occurs in information-retrieval functions (e.g., search_recipes() in Figure 1 (b)). Accordingly, we first assign each complexity type to the functions most likely to encounter this type of complexity in the real world; and then inject these complexities by modifying the corresponding API implementations. Our evaluation on WildAgtEval shows that most complexity scenarios consistently degrade performance across strong LLM agents (e.g., Claude-4-Sonnet Anthropic 2025c ), with irrelevant information complexity posing the greatest challenge, causing an average performance drop of 27.3%. Moreover, when multiple complexities accumulate, the performance degrades by up to 63.2%. Qualitative analysis further reveals that, when facing unresolvable tasks, LLMs persist in attempting to solve them, ultimately distorting user intent and producing misleading success responses. 2 Related Work 2.1 API-Based Benchmarks for LLM Agents Existing API-based benchmarks have advanced agent evaluation by focusing on multi-step reasoning through functional dependencies. BFCLv3 Patil et al. 2025 and \tau -bench Yao et al. 2024 ; Barres et al. 2025 introduce sequential dependency scenarios (e.g., search_media() → play() for playback). Moreover, ComplexFuncBench Zhong et al. 2025 incorporates user constraints into multi-step tasks and NESTful Basu et al. 2024 extends to mathematical domains. Meanwhile, Incomplete-APIBank Yang et al. 2024 and ToolSandbox Lu et al. 2024 assess robustness of agents to missing APIs and service state variations, respectively. Nevertheless, prior benchmarks still underrepresent real-world API complexities (Table 1 ), yielding overly optimistic assessments; this work addresses this gap by integrating such complexities into agent evaluation. 2.2 User-Based Benchmarks for LLM Agents User-based benchmarks assess whether LLM agents fulfill diverse real-world user instructions of varying complexity. MT-Eval Kwan et al. 2024 , Multi-Turn-Instruct Han 2025 , and Lost-In-Conv Laban et al. 2025 evaluate how well agents handle diverse user instructions in multi-turn interactions, emphasizing challenges such as intent disambiguation, multi-intent planning, and preference elicitation. IHEval Zhang et al. 2025 and UserBench Qian et al. 2025 further extend these evaluations by incorporating scenarios where