In-N-Out: A Parameter-Level API Graph Dataset for Tool Agents

Seungkyu Lee Nalim Kim Yohan Jo · 2025

Read In-N-Out: A Parameter-Level API Graph Dataset for Tool Agents by Seungkyu Lee Nalim Kim Yohan Jo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-01560.txt.

Introduction

Tool agents, language model systems that call outside software interfaces, often stumble on complex tasks that need several calls in the right order. Seungkyu Lee, Nalim Kim, and Yohan Jo investigate turning interface documentation into a structured graph that captures how the interfaces depend on each other, at the level of their parameters. Such a graph could guide agents through multi-tool queries.

The team introduces In-N-Out, the first expert-annotated data set of such interface graphs, built from two real-world benchmarks and their documentation. Using the graphs nearly doubled performance on tool retrieval and multi-tool query generation, compared with documentation alone. Models fine-tuned on the data set to build graphs themselves closed ninety percent of that gap. Lee and the co-authors release the data set as a resource for tool agent research.

Catalog record

CREATORSeungkyu Lee Nalim Kim Yohan Jo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE52 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-01560

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

In-N-Out: A Parameter-Level API Graph Dataset for Tool Agents Seungkyu Lee Nalim Kim Yohan Jo Abstract Tool agents—LLM-based systems that interact with external APIs—offer a way to execute real-world tasks. However, as tasks become increasingly complex, these agents struggle to identify and call the correct APIs in the proper order. To tackle this problem, we investigate converting API documentation into a structured API graph that captures API dependencies and leveraging it for multi-tool queries that require compositional API calls. To support this, we introduce In-N-Out , the first expert-annotated dataset of API graphs built from two real-world API benchmarks and their documentation. Using In-N-Out significantly improves performance on both tool retrieval and multi-tool query generation, nearly doubling that of LLMs using documentation alone. Moreover, graphs generated by models fine-tuned on In-N-Out close 90% of this gap, showing that our dataset helps models learn to comprehend API documentation and parameter relationships. Our findings highlight the promise of using explicit API graphs for tool agents and the utility of In-N-Out as a valuable resource. We will release the dataset and code publicly. 1 Introduction Recent advances in Large Language Models (LLMs) have spurred growing interest in building tool agents —LLM-based systems that interact with external APIs to execute real-world tasks Schick et al. 2023 ; Shen et al. 2023 . Unlike traditional language models that rely solely on pre-trained knowledge, tool agents can retrieve real-time information, access databases, and operate services via API calls Song et al. 2023 . As user queries grow more complex, solving them often involves chaining multiple APIs—commonly known as multi-tool queries —where one API call requires the outputs produced by a previous call. This makes it crucial for tool agents not only to understand individual APIs but also to capture interdependencies among APIs to plan valid call sequences Lumer et al. 2025 ; Zhang et al. 2024 . Figure 1: Illustration of how parameter-level API graphs support (a) Tool Retrieval , where the agent identifies prerequisite APIs needed to supply inputs for a target API (e.g., FollowArtist), and (b) Multi-Tool Query Generation , where the agent selects interdependent APIs to construct coherent multi-tool queries. A concrete example in Figure 1 a illustrates why such interdependencies are essential. To fulfill the query “Follow all the EDM artists on Spotify that have at least 1K followers” , an agent must first call other APIs to obtain an access token and artist IDs, before invoking FollowArtist(). Without explicit knowledge of parameter connections, agents often fail to chain these APIs correctly, as the required dependencies are obscured among the large number of available APIs. While prior studies have explored training tool agents on large datasets of multi-tool queries Qin et al. 2024 ; Shim et al. 2025 , a more natural and general solution, akin to what human developers do, is to use API documentation to identify API dependencies. However, since LLMs struggle to comprehend parameter-level dependencies from noisy, real-world API documentation (§ 4 ), we focus on a relatively underexplored approach: converting API documentation into an explicit API graph, enabling LLMs to use this structured information. Some prior studies that have attempted to use API graphs have focused primarily on limited domains Wang et al. 2024 or automatic graph construction with heuristics Liu et al. 2024b ; Shen et al. 2024 , leaving the true potential of accurate API graphs in real-world scenarios unexplored. Since manual graph construction is not scalable for arbitrary API sets, a high-quality dataset is needed to train and evaluate a documentation-to-graph conversion module and ensure its generalizability to unseen APIs. Our approach is arguably more generalizable and scalable than simply training a tool agent on large multi-tool tasks, and the graphs can even facilitate the generation of multi-tool queries by providing the underlying API composition for complex queries (Figure 1 b). Against this backdrop, we introduce In-N-Out , an API graph dataset constructed through expert annotation. In-N-Out represents both APIs and their parameters as nodes, with directed edges indicating when an output from one API can serve as a valid input to another. The APIs are sourced from AppWorld Trivedi et al. 2024 and NESTful Basu et al. 2025 , spanning 550 APIs, and over 30,000 parameter-level edges, thus covering diverse and realistic tool-use scenarios. We first demonstrate that training LLMs on our In-N-Out dataset improves their ability to construct parameter-level API graphs from noisy documentation, even for unseen APIs. Specifically, we prompt models to decide whether a connection exists between the parameters of two given APIs. Zero-shot models perform poorly on this task, indicating that LLM-based automatic graph construction is unreliable. In contrast, models fine-tuned on In-N-Out achieve significantly higher accuracy and generalize well to unseen APIs and datasets, sometimes reaching the accuracy achieved by training on the test dataset itself. This demonstrates that In-N-Out allows LLMs to learn the underlying ability to comprehend API documentation and parameter relationships. We further draw two key insights from experiments on two tasks: (1) solving multi-tool queries and (2) generating multi-tool queries (which may be used for training and benchmarking tool agents). First, using In-N-Out significantly improves performance on both tasks compared to using documentation alone. This justifies our approach of using explicit, high-quality API graphs for overcoming challenges in tool agents. Second, even automatically generated graphs from models fine-tuned on In-N-Out yield substantial gains, closing over 90% of the performance gap on both tasks compared to using documentation directly. The results demonstrate that In-N-Out can serve as a valuable resource for graph-based approaches for tool agents. In summary, our contributions are as follows: • We construct and release In-N-Out, an API graph dataset annotated by experts, capturing parameter-level dependencies between real-world APIs. • We show that fine-tuning on In-N-Out enables LLMs to infer parameter-level connections from textual API documentation and generalize to unseen APIs. • We demonstrate that In-N-Out substantially improve tool agent performance, and that even automatically generated graphs reduce the performance gap relative to human-labeled graphs. 2 Related Work Developing tool agents has centered on two key tasks: tool retrieval and planning (§ 2.1 ), and multi-tool query generation (§ 2.2 ). Graph-based approaches have supported both tasks by capturing API relationships and enabling scalable coordination. 2.1 Tool Retrieval and Planning As real-world tasks increasingly require multiple APIs in sequence, both accurate retrieval and correct input provision become essential ( Patil et al. 2025 ; Xu et al. 2024 ; Shi et al. 2025 ; Du et al. 2024 ; Lumer et al. 2024 ) . To support these capabilities, previous work has explored several approaches: refining API documentation to make it more interpretable for LLMs ( Hsieh et al. 2023 ; Yuan et al. 2025 ; Chen et al. 2024 ) , fine-tuning LLMs on query-API call pairs ( Qin et al. 2024 ; Patil et al. 2024 ) , and learning tool-specific embeddings ( Hao et al. 2023 ; Wang et al. 2025 ) . However, these strategies often fail to capture the underlying dependencies among APIs, while also requiring high training costs and generalizing poorly to unseen APIs. To address this, recent studies have explored graph-based approaches that represent API relationships explicitly to support tool retrieval ( Liu et al. 2024b

Read the complete text