Synthesize and Reward — Reinforcement Learning for Multi-Step Tool Use in Live Environments
Ibrahim Abdelaziz, Asim Munawar, Kinjal Basu, Maxwell Crouse, · 2026
Read Synthesize and Reward — Reinforcement Learning for Multi-Step Tool Use in Live Environments by Ibrahim Abdelaziz, Asim Munawar, Kinjal Basu, Maxwell Crouse, (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03892.txt.
Introduction
Abdelaziz and co-authors train language models for multi-step tool use inside live environments. Three obstacles block such training. Stateful execution environments are costly to build. Synthetic queries often miss the server's real state, so calls fail. Recall-based rewards push verbose tool calling. PROVE answers all three obstacles together.
A library of 20 stateful MCP servers exposes 343 tools for live training. Session-scoped state keeps runs isolated. A state-machine pipeline synthesizes multi-turn trajectories grounded in live-sampled state. Generated queries thus reference entities that exist. A programmatic reward adds an adaptive efficiency penalty against verbosity. Four models train with GRPO on about 13,000 examples. Gains reach about ten points on BFCL Multi-Turn, with solid rises on two other suites. Tool orchestration improves across two model families.
Catalog record
| CREATOR | Ibrahim Abdelaziz, Asim Munawar, Kinjal Basu, Maxwell Crouse, |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.03892
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Synthesize and Reward — Reinforcement Learning for Multi-Step Tool Use in Live Environments Ibrahim Abdelaziz, Asim Munawar, Kinjal Basu, Maxwell Crouse, Abstract Training LLMs to orchestrate multi-step tool calls is held back by three coupled obstacles: realistic stateful execution environments are costly to build, synthetic training queries are often detached from the server’s actual state (so the generated tool calls fail to execute), and recall-based RL rewards incentivize verbose tool-calling patterns. We present Prove (Programmatic Rewards On Verified Environments), a framework with three contributions: (1) a library of 20 stateful MCP (Model Context Protocol) servers exposing 343 tools, enabling live-execution RL training with session-scoped state isolation; (2) a state-machine data synthesis pipeline that generates multi-turn tool-call trajectories grounded in live-sampled server state , so generated queries reference entities that actually exist; and (3) a multi-component programmatic reward with an adaptive efficiency penalty that counters the verbosity incentive of recall-based rewards. We train four models (Qwen3-4B, Qwen3-8B, Qwen2.5-7B, Granite-4.1-8B) with GRPO on the resulting ∼ \sim 13K training examples. On BFCL Multi-Turn, τ 2 \tau^{2} -bench, and T-Eval, Prove yields improvements of up to + 10.2 +10.2 , + 6.8 +6.8 , and + 6.5 +6.5 points respectively, demonstrating that this framework yields consistent gains on multi-step tool orchestration across two model families. 1 Introduction Training LLMs to orchestrate multi-step tool calls—selecting the correct API, filling parameters from context, and respecting inter-call data dependencies—is a prerequisite for deploying autonomous agents in production ( 20 ; 16 ) . The problem is harder than single-turn function calling: the model must propagate intermediate results, handle missing information gracefully, and abstain when no tool is applicable. Two complementary lines of work have emerged in response. Data synthesis and supervised fine-tuning. xLAM ( 25 ) releases a large corpus of verified single-turn function-calling traces, and APIGen-MT ( 11 ) extends this to multi-turn trajectories. More recent pipelines target realism and scale for multi-turn tool use: Magnet ( 21 ) composes traces from tool-dependency graphs, ToolACE-MT ( 23 ) introduces non-autoregressive multi-turn generation, and TOUCAN ( 19 ) scales synthesis to 1.5M traces against real MCP servers. These pipelines typically feed a separate SFT stage and are not coupled to execution feedback during training. Reinforcement learning for tool use. A parallel line uses RL to shape tool-use behavior from reward signals without large SFT corpora. ToolRL ( 3 ) , ReTool ( 7 ) , StepTool ( 22 ) , Tool-R1 ( 27 ) , Nemotron-Research-Tool-N1 ( 26 ) , MUA-RL ( 28 ) , and Tool Zero ( 24 ) explore per-call and step-grained rewards, multi-turn user simulation, and RL-from-scratch. Closest to our setting, 5 decomposes the reward into per-call validity and sequence-level consistency on a single cached-API domain, while the concurrent AgenticQwen ( 14 ) scales to \sim 100K branching trajectories with LLM-as-judge rewards (Qwen3-235B). Despite this progress, three gaps remain unaddressed jointly: (i) most pipelines execute against cached or simulated APIs, missing the state-dependent failure modes that arise in live, stateful services; (ii) synthetic queries are often detached from the server’s actual state, producing traces whose parameters do not correspond to any real entity; and (iii) sequence-level rewards based purely on recall incentivize verbosity —RL-trained models emit excess tool calls to maximize GT coverage, a regression that is rarely penalized explicitly. We close these gaps with Prove (Programmatic Rewards On Verified Environments), 1 1 1 Code, the 20 MCP server library (343 tools), synthesized training data, and trained model checkpoints will be released soon. a framework that tightly couples data synthesis to the RL loop through a shared live-execution backend: 1. Live MCP environments. A library of 20 stateful Model Context Protocol servers exposing 343 tools, with session-scoped state isolation, used for both data synthesis and RL training. Unlike cached APIs, these capture realistic state-dependent execution dynamics. 2. Grounded state-machine data synthesis. A pipeline that probes each live server for real entities, grounds query generation in that sampled state, drives a state-machine teacher through multi-turn conversations against the server, and replay-validates every trace. This produces \sim 13K training examples (multi-turn MCP conversations, missing-function clarification, and abstention) without manual annotation or an LLM-as-judge. 3. Programmatic multi-component reward. A five-part reward decomposing tool-use quality into validity, dependency-ordered coverage, an adaptive efficiency penalty with a complexity-scaled call budget, a tool-name signal, and an argument-value matching bonus—fully programmatic with no external judge model (§ 3.3 ). An ablation (§ 4.5 ) shows that the adaptive budget and the tool-name signal are the two most load-bearing components. Table 1 summarizes results across four models from two architecture families, trained with identical hyperparameters. Table 1 : Improvement over base model ( \Delta ) after training with Prove . All models trained for 350 GRPO steps. Model BFCL MT \boldsymbol{\Delta} TAU2 \boldsymbol{\Delta} T-Eval \boldsymbol{\Delta} Qwen3-4B 29.0 +10.2 28.6 +3.6 78.2 +1.5 Qwen3-8B 26.9 +0.9 29.3 +3.5 80.7 +1.9 Qwen2.5-7B 16.8 +3.7 21.8 +6.8 73.4 +6.5 Granite-4.1-8B 33.5 +0.1 31.3 +6.4 77.9 +4.4 We evaluate on BFCL Multi-Turn ( 10 ) , \tau^{2} -bench ( 2 ) , and T-Eval ( 4 ) . Prove improves all four models on all three benchmarks, using \sim 13K training examples, roughly 8\times less than concurrent RL pipelines such as AgenticQwen ( 14 ) , and no judge model. Figure 1 illustrates how the system works. Figure 1 : System overview. An LLM teacher generates multi-turn trajectories against live MCP servers using auto-discovered dependency graphs. The GRPO trainer rolls out against the same servers and computes the five-component reward. Session-scoped state isolation ensures reproducibility. 2 Related Work Data Synthesis and Supervised Fine-Tuning for Tool Use. A growing body of work scales synthetic data to train tool-using LLMs via supervised fine-tuning. xLAM ( 25 ) releases a large corpus of verified single-turn function-calling traces, and APIGen-MT ( 11 ) extends this to multi-turn trajectories with stronger verification. Magnet ( 21 ) synthesizes multi-turn traces by translating a tool-dependency graph into natural conversations and distilling them into smaller models. ToolACE-MT ( 23 ) proposes non-autoregressive generation for agentic multi-turn interactions, and TOUCAN ( 19 ) scales synthesis to 1.5M traces from real-world MCP servers. Closer to our setting, 6 address the challenge of simulating complex multi-turn interactions when the execution backend is stateless. These pipelines improve trajectory quality but typically feed a separate SFT stage and do not train against live execution feedback. Our pipeline differs in three ways: (i) it is driven by a state machine that grounds queries in live server state at generation time, (ii) it replay-validates every conversation against a freshly reset environment, and (iii) it produces \sim 13K examples that feed directly into RL rollouts rather than an intermediate SFT corpus. Reinforcement Learning for Tool Use. Reinforcement learning has become a popular way to elicit tool-use behavior without large SFT corpora. ToolRL ( 3 ) showed that reward signals alone can teach tool-use capabilities, and ReTool ( 7 ) combined reasoning and tool use through strategic RL. StepTool ( 22 ) introduces step-grained rewards for multi-step tool chains, while Nemotron-Research-Tool-N1 ( 26 ) explores reinforced reasoning for tool-using models. Tool-R1 ( 27 ) targets sample efficiency with compositional multi-step rewards, and MUA-RL ( 28 ) trains multi-turn policies against an LLM