AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios
Kou Shi, Ziao Zhang, Shiting Huang, Avery Nie, Zhen Fang, Qiuchen Wang, Lin Chen, Huaian Chen, Zehui Chen, Feng Zhao · 2026
Read AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios by Kou Shi, Ziao Zhang, Shiting Huang, Avery Nie, Zhen Fang, Qiuchen Wang, Lin Chen, Huaian Chen, Zehui Chen, Feng Zhao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-27995.txt.
Introduction
Shi and colleagues test whether AI agents can juggle several tool-using tasks at once while waiting on slow tool replies. Their AsyncTool benchmark runs varied tasks concurrently with realistic delays, scoring coordination as well as completion. Delayed feedback degraded current agents markedly, with the best coordinators winning on task switching, dependency tracking, and state upkeep.
Catalog record
| CREATOR | Kou Shi, Ziao Zhang, Shiting Huang, Avery Nie, Zhen Fang, Qiuchen Wang, Lin Chen, Huaian Chen, Zehui Chen, Feng Zhao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 67101 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.27995
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AsyncTool: Evaluating the Asynchronous Function Calling Capability under Multi-Task Scenarios Kou Shi, Ziao Zhang, Shiting Huang, Avery Nie, Zhen Fang, Qiuchen Wang, Lin Chen, Huaian Chen, Zehui Chen, Feng Zhao Abstract Large language model (LLM)-based agents have demonstrated strong capabilities in leveraging external tools to solve complex tasks. However, existing evaluations largely overlook the temporal dimension of tool invocation, particularly the impact of tool response latency, and are typically limited to single-task settings. In real-world applications, multiple tasks often need to be executed concurrently, and overall efficiency critically depends on whether an agent can utilize idle time while waiting for tool responses. We refer to this capability as asynchronous tool calling. To evaluate this capability, we propose AsyncTool, a benchmark for assessing LLM-based agents in interactive multi-task tool-use environments with delayed tool feedback. AsyncTool presents multiple heterogeneous tasks simultaneously and simulates realistic tool response latency during execution. Using a hybrid data evolution strategy, we construct a diverse asynchronous multitasking dataset covering multiple scenarios and tool-use patterns. We evaluate models at three levels—step, sub-task, and task—and further introduce efficiency-oriented metrics to measure task coordination and completion efficiency. Extensive experiments show that delayed tool feedback poses substantial challenges to current agents, leading to clear performance degradation. Models that better coordinate task switching and dependency tracking tend to achieve stronger performance on AsyncTool.Our analysis identifies the main failure modes of current tool agents and provides practical guidelines for designing future systems with stronger temporal reasoning and coordination capabilities.The code is available at https://github.com/StoKou/repo-asynctool 1 Introduction Recent advances in large language models (LLMs) have significantly improved their ability to follow instructions and understand context, leading to increasingly capable LLM-based agents for tool use ( 28 ; 12 ; 3 ; 44 ; 33 ; 50 ; 7 ; 37 ; 10 ; 21 ; 53 ) . This capability enables them to handle more sophisticated, multi-step tasks that require external information or actions, and to achieve strong performance across diverse tool-use scenarios ( 26 ; 23 ; 6 ) . However, real-world environments are often more complex, frequently requiring the concurrent execution of multiple tasks that may involve different tools. In practical settings, function calls usually incur latency, and executing tasks sequentially in a synchronous manner fails to fully utilize idle waiting time, thereby reducing overall efficiency. To better evaluate and enhance the agent’s performance under such conditions, we introduce the concept of Asynchronous Tool Call into the interaction between the agent and the environment, where the agent should utilize these idle intervals to advance other available tasks. Motivated by these gaps, we identify three critical observations: (i) Inadequate evaluation of the agent’s capability to complete multiple tasks in asynchronous scenarios. Existing studies are typically restricted to single-task scenarios in which tools operate in an immediate response manner ( 56 ; 31 ; 42 ; 17 ; 30 ; 46 ) , overlooking the evaluation for multiple tasks in asynchronous scenarios. (ii) Lack of alignment with real-world conditions in interactive environments involving real-time tool calls. Existing asynchronous planning benchmarks do not operate within interactive environments, which is inconsistent with real-world scenarios involving real-time tool calls ( 24 ) . (iii) Insufficient metrics and standardized protocols specific to concurrent tasks with delayed and out-of-order tool feedback. Traditional benchmarks involving time delays do not cover tool-using tasks and cannot be transferred to agentic tasks ( 51 ; 16 ) . To bridge these gaps, we propose AsyncTool , a benchmark for evaluating the ability of LLM-based agents to perform asynchronous tool calling in interactive multi-task scenarios. To our knowledge, AsyncTool is the first benchmark that jointly considers delayed tool feedback, concurrent multi-task execution, multi-step function calling, and dependency-aware task coordination. (i) Our benchmark consists of combinations of multiple tasks, where each task contains intra-task step dependencies and different tasks can be pursued concurrently. This design allows an agent to use the waiting periods caused by tool latency to advance other independent tasks. (ii) To better approximate real-world tool-use conditions, we simulate tool-specific response latencies, integrate multiple tasks into a shared interaction process, and require the agent to make progress on them through asynchronous function calls. This setting provides a practical environment for assessing whether agents can coordinate multiple tasks under delayed and potentially out-of-order tool feedback. Table 2 compares AsyncTool with existing benchmarks on tool calling and asynchronous execution. (iii) To comprehensively evaluate asynchronous tool-use capabilities, we assess model performance at three levels: Step Level, Sub-Task Level, and Task Level, covering fine-grained tool-call correctness, intermediate subtask completion, and end-to-end multi-task success. In addition, we introduce efficiency-oriented metrics to measure task-interleaving behavior and completion efficiency under tool latency. Through extensive experiments, we find that delayed tool feedback poses substantial challenges to current LLM-based agents, especially in maintaining task states before results arrive. Compared with synchronous or immediate-response settings, asynchronous execution leads to clear performance degradation, especially when models prematurely continue a task before its dependent tool result has returned. Our analysis further shows that effective asynchronous tool use requires more than frequent task switching: models must coordinate task switching with dependency tracking and state maintenance. Stronger models are better able to utilize idle waiting periods to advance other tasks while resuming pending tasks at the appropriate time, whereas weaker models often suffer from dependency violations, task neglect, and tool confusion. These findings highlight the importance of temporal coordination for future tool-using agents. The main contributions of our work are summarized as follows: • We propose AsyncTool , a benchmark for evaluating asynchronous tool calling in interactive multi-task environments with delayed tool feedback. • We construct a diverse asynchronous multitasking dataset by composing validated single-task tool-use trajectories through a hybrid data-evolution strategy. The resulting tasks cover different task numbers, task types, scenarios, and dependency structures. • We design a multi-level evaluation protocol that assesses model performance at the step, sub-task, and task levels, capturing both fine-grained tool-call correctness and end-to-end task completion. • We introduce efficiency-oriented metrics to analyze task interleaving and completion behavior under tool latency, and conduct extensive experiments to reveal the challenges current LLM agents face in temporal coordination and dependency tracking. Figure 1: Overview of the dataset construction process. The pipeline starts by collecting raw data from tool-use benchmarks, categorizing it by scenario. Task-step dependencies are reinforced, and execution trajectories are reconstructed using Gemini 2.5 Pro, followed by manual verification for accuracy and determinism. Finally, data evolution occurs through hybrid strategies, with filtering producing the final multitasking dataset. 2 AsyncTool Building on the motivation introduced in Section 1 , AsyncTool simulates tool response latency and evaluates asynchronous tool calling in multi-task settings. This section first formalizes the interaction paradigm in AsyncTool , where agents must coordinate multiple tasks under delayed tool feedback. We then describe the asynchronous multitasking dataset and evaluation protocol. 2.1 Agent as a Concurrent Tool-Using System While agents’ ability to solve problems through tool use is well established, their practical effectiveness can be limited by the non-negligible response latencies of tool calls in real-world scenarios. This raises an important question: can agents use idle time from pending tool calls to work on other tasks? AsyncTool studies this by simulating delayed tool feedback and concurrent execution. Unlike standard tool-use settings, tool results