Procedural Environment Generation for Tool-Use Agents
Michael Sullivan, Mareike Hartmann, Alexander Koller · 2025
Read Procedural Environment Generation for Tool-Use Agents by Michael Sullivan, Mareike Hartmann, Alexander Koller (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-11045.txt.
Introduction
Michael Sullivan, Mareike Hartmann, and Alexander Koller introduce in 2025 RandomWorld, a generator of play worlds for training tool-use agents. Good tool training data are scarce, above all for live online reward training. Past fake data tend to be static or lack joined steps.
RandomWorld builds interactive tools and layered task data by procedure. Models tuned and reward-trained on the fake data gain on a range of tool benchmarks. Two scores on one data set hit new best marks after training. More fake data brings more gains in the scaling tests, which hints that full fake training diets may work.
Catalog record
| CREATOR | Michael Sullivan, Mareike Hartmann, Alexander Koller |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 55 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-11045
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Procedural Environment Generation for Tool-Use Agents Michael Sullivan, Mareike Hartmann, Alexander Koller Abstract Although the power of LLM tool-use agents has ignited a flurry of recent research in this area, the curation of tool-use training data remains an open problem—especially for online RL training. Existing approaches to synthetic tool-use data generation tend to be non-interactive, and/or non-compositional. We introduce RandomWorld, a pipeline for the procedural generation of interactive tools and compositional tool-use data. We show that models tuned via SFT and RL on synthetic RandomWorld data improve on a range of tool-use benchmarks, and set the new SoTA for two metrics on the NESTFUL dataset. Further experiments show that downstream performance scales with the amount of RandomWorld-generated training data, opening up the possibility of further improvement through the use of entirely synthetic data. 1 Introduction A substantial amount of current research has focused on equipping LLMs with the means to employ tools—external functions (e.g. APIs) that provide the LLM with enhanced knowledge and/or capabilities—with the goal of achieving AI assistants capable of executing complex sequences of tool calls to accomplish tasks such as information retrieval ( Zheng et al. 2024 ; Li et al. 2025 ; Zheng et al. 2025 , e.g.) , making online purchases ( Yao et al. 2022 ; Cai et al. 2025 , e.g.) , editing a user’s local files ( Trivedi et al. 2024 , e.g.) , etc. In line with broader findings that LLMs post-trained via reinforcement learining (RL) exhibit superior generalization capabilities to supervised-fine-tuned (SFT) agents ( Chu et al. 2025 ) , recent work demonstrates that LLMs fine-tuned for tool use through online RL can better adapt to tools and tasks not seen during training ( Qian et al. 2025 ; Feng et al. 2025 ) . Figure 1: Example of an agent executing a (simple) non-linear, compositional task with interactive tools in a RandomWorld environment, with an illustration of the pipleine components that go into the generation of the task and environment. To enable the agent to effectively employ tools across a wide range of tasks and domains, LLM tool-use training necessitates a sufficiently rich environment. However, dataset curation for online RL tool-use training remains a major challenge: it is not feasible to directly train tool use agents in the real-world (due to the obvious dangers of such an approach), and hand-crafting simulated APIs—and tasks using these APIs—is a time-consuming process that is not feasible at the scale required for (SFT or RL) LLM fine-tuning. As a result, many existing datasets are comprised of either small or non-interactive (i.e. not callable) tool inventories, and/or non-compositional tasks (see Section 2 ). While training on such datasets may be sufficient to conquer simpler benchmarks (e.g. Yan et al. 2024 ; Patil et al. 2024 , etc.; see Section 2 ) , they fail to satisfactorily improve model performance on benchmarks with more complex tasks: namely, tasks necessitating the non-linear chaining of tool calls in an interactive setting (e.g. Zhuang et al. 2023 ; Basu et al. 2024 ; Trivedi et al. 2024 , etc.; see Sections 2 and 4 ) . On the other hand, those datasets that do contain large, interactive tool inventories with compositional tasks are carefully hand-crafted ( Trivedi et al. 2024 , e.g.) , and so contain a very limited number of tasks: while valuable as benchmarks, these datasets are not suitable for large-scale RL training. In this paper, we introduce RandomWorld (Section 3 ), a pipeline for the procedural generation of tools and tasks, that is capable of generating a virtually unlimited amount of training data for online RL (or SFT) training of tool-use agents. As illustrated in Figure 1 , RandomWorld employs a fine-grained type hierarchy to generate non-linear DAGs (trajectory skeletons) of tool calls from the signatures of fully-interactive (i.e. callable) synthetic tools (see Section 3 ). These trajectory skeletons parameterize the construction of environments: data structures consisting of tools, a goal state (value to return), and an instruction. This procedural generation pipeline results in environments that have: i. Depth (of the tool inventory): a large toolset across a diverse assortment of domains, to facilitate the agent’s ability to generalize to unseen tools. ii. (Non-linear) Compositionality : chainable tools—and objectives necessitating non-linear tool-chaining—to emulate complex, real-world tasks (see e.g. Figure 1 ). iii. Interactivity : intermediate tool outputs that are visible to the agent, allowing the inspection of outputs and (if necessary) correction of the tool-call sequence—as is possible in many real-world settings. Models fine-tuned with RandomWorld—through either RL or SFT—exhibit increased performance on various tool-use benchmarks (Section 4 ), and set the new SoTA on two NESTFUL Basu et al. 2024 metrics. Further experiments show that downstream performance scales with the size of the tool inventory and number of tasks in the training set (Section 5 ), indicating that further training with RandomWorld can further improve performance without the need for costly human annotation. We release all code for the RandomWorld pipeline and the experiments conducted in this paper available on GitHub 1 1 1 https://github.com/coli-saar/randomworld . 2 Related Work Non-Interactive Tool-Use Datasets. Although many existing tool use datasets have large API inventories and complex tasks that require non-linear tool chaining, they do not include interactive tools (see Table 1 ). For example, while APIGen Liu et al. 2024b employs real-world, callable APIs during dataset generation, this pipeline only creates non-interactive SFT ( Rafailov et al. 2023 , and similar, e.g. DPO;) training data: namely, queries/instructions annotated with solution paths. Similarly, the ToolBench Qin et al. 2024 dataset contains an interactive test set, but a non-interactive train set—while it may be theoretically possible to adapt ToolBench to online RL training, the latency accompanying this dataset’s use of real-world APIs would impede such an approach. In the ToolACE Liu et al. 2024a and APIBank Li et al. 2023 training sets, both the tasks/solution paths and tools are synthesized, as in RandomWorld. Unlike RandomWorld, the tools in ToolACE and APIBank are not callable: an LLM simulates outputs during dataset creation. This again results in SFT-only training sets. The UltraTool dataset Huang et al. 2024 contains LLM-synthesized tasks and solution paths that require the complex, non-linear compositional use of over 2,000 tools. However, these tools are not functional, and so are not interactive. While APIBench Patil et al. 2024 has a deep tool inventory—over 700 real-world APIs—it is not interactive or compositional: the LLM-synthesized tasks in this dataset require only one API call. The non-interactivity of the tools in these datasets impairs evaluation, which further hinders their employment for online RL training, as evaluation and reward are inextricably linked. When tools are not interactive, scores must be assigned through either exact match of the tool call sequence—which ignores the possibility of a task having more than one solution path—or LLM-as-a-judge evaluation. Interactive Tool-Use Datasets and Benchmarks. On the other hand, the WebShop Yao et al. 2022 dataset is compositional and fully interactive. However, the WebShop tool inventory is shallow (eight actions/tools), which limits the knowledge obtained from training on the dataset to this narrow domain. Similarly, the Berkeley Function-Calling Leaderboard ( Yan et al. 2024 , BFCL V3;) contains fully interactive APIs. As in RandomWorld, BFCL V3 task