MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use

Weikang Zhao 1,2,* Xili Wang 1,3,*, † \dagger Chengdi Ma 3 Lingbin Kong 1 Zhaohua Yang 1 Mingxiang Tuo 1 Xiaowei Shi 1 Yitao Zhai 1, † \dagger Xunliang Cai 1 1 Meituan Inc. 2 Institute of Automation, Chinese Academy of Sciences 3 Peking University [email protected] {xiliwang,mcd2020}@stu.pku. · 2025

Read MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use by Weikang Zhao 1,2,* Xili Wang 1,3,*, † \dagger Chengdi Ma 3 Lingbin Kong 1 Zhaohua Yang 1 Mingxiang Tuo 1 Xiaowei Shi 1 Yitao Zhai 1, † \dagger Xunliang Cai 1 1 Meituan Inc. 2 Institute of Automation, Chinese Academy of Sciences 3 Peking University [email protected] {xiliwang,mcd2020}@stu.pku. (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18669.txt.

Introduction

Weikang Zhao and eight co-authors present a 2025 arXiv paper on training AI agents that chat with users while calling tools. Real users change their minds mid-task, so an agent must keep clarifying needs as it books, buys, or fixes. Past training used frozen scripts, leaving agents brittle with live, shifting demands.

The team's framework puts a simulated user, played by a language model, inside the training loop itself. The agent learns by practice across many turns, earning rewards for both talk and tool use. Their trained model scored at or above far larger open models on retail, airline, and telecom test suites. It also led on two other tool-use benchmarks. The authors claim this is the first such user-in-the-loop training for tool agents, and a step toward assistants that cope with real conversation.

Catalog record

CREATORWeikang Zhao 1,2,* Xili Wang 1,3,*, † \dagger Chengdi Ma 3 Lingbin Kong 1 Zhaohua Yang 1 Mingxiang Tuo 1 Xiaowei Shi 1 Yitao Zhai 1, † \dagger Xunliang Cai 1 1 Meituan Inc. 2 Institute of Automation, Chinese Academy of Sciences 3 Peking University [email protected] {xiliwang,mcd2020}@stu.pku.
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-18669

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MUA-RL: Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use Weikang Zhao 1,2,* Xili Wang 1,3,*, † \dagger Chengdi Ma 3 Lingbin Kong 1 Zhaohua Yang 1 Mingxiang Tuo 1 Xiaowei Shi 1 Yitao Zhai 1, † \dagger Xunliang Cai 1 1 Meituan Inc. 2 Institute of Automation, Chinese Academy of Sciences 3 Peking University [email protected] {xiliwang,mcd2020}@stu.pku.edu.cn {konglingbin,yangzhaohua02,tuomingxiang02,shixiaowei02,zhaiyitao,caixunliang}@meituan.com Abstract With the recent rapid advancement of Agentic Intelligence, agentic tool use in LLMs has become increasingly important. During multi-turn interactions between agents and users, the dynamic, uncertain, and stochastic nature of user demands poses significant challenges to the agent’s tool invocation capabilities. Agents are no longer expected to simply call tools to deliver a result; rather, they must iteratively refine their understanding of user needs through communication while simultaneously invoking tools to resolve user queries. Existing reinforcement learning (RL) approaches for tool use lack the integration of genuinely dynamic users during the RL training process. To bridge this gap, we introduce MUA-RL (Multi-turn User-interacting Agent Reinforcement Learning for agentic tool use), a novel reinforcement learning framework that, for the first time in the field of agentic tool use, integrates LLM-simulated users into the reinforcement learning loop. MUA-RL aims to enable autonomous learning of models to communicate with users efficiently and use various tools to solve practical problems in dynamic multi-turn interactions. Evaluations are done on several multi-turn tool-using benchmarks (see Figure 1 ). Specifically, MUA-RL-32B achieves 67.3 on TAU2 Retail, 45.4 on TAU2 Airline, 28.3 on TAU2 Telecom, 28.4 on BFCL-V3 Multi Turn, and 82.5 on ACEBench Agent – outperforming or matching the performance of larger open-source models such as DeepSeek-V3-0324 and Qwen3-235B-A22B in non-thinking settings. K eywords Large Language Models \cdot Agentic Tool Use \cdot Reinforcement Learing \cdot Multi-turn User-interacting Figure 1: MUA-RL main results. 1 Introduction The paradigm of Large Language Models (LLMs) is shifting towards Agentic Intelligence – models are now equipped with extensive external tools to interact dynamically with users and the world. This shift signifies a move away from static imitation learning, paving the way for models that engage in active learning through interactions, going beyond their pre-training and post-training data. Consequently, this new paradigm demands more on the model’s ability to use tools effectively in multi-turn interactions and communicate clearly with users. While current LLMs mainly undergo supervised fine-tuning (SFT) on synthesized tool-using data to acquire capabilities to interact with both users and the world, reinforcement learning (RL) is believed to have better generalization than SFT [ 1 ] . Combined with RL, Large language models (LLMs) have shown remarkable progress, particularly in static, well-defined domains. For example, DeepSeek-Math [ 2 ] achieves strong performance in mathematical problem solving. Recent works have begun to incorporate interactions with external environments into the reinforcement learning process of LLMs. Retool [ 3 ] enhances RL by integrating code interpreter to enable LLMs to interact with a real code sandbox, SkyRL [ 4 ] facilitates interactions between LLMs and actual Docker environments during RL training, and RAGEN [ 5 ] depends on symbolic or pre-specified environments. However, these systems generally operate within predetermined environments and rely on pre-scripted queries. As a result, current RL approaches may struggle on the dynamics and unpredictability of real-world interactions with users. In real-world scenarios, users’ needs are highly dynamic and unpredictable. Users often adjust their questions and expectations based on the model’s responses, creating a feedback loop that requires continuous bidirectional adaptation. This interactive and co-evolutionary dynamic is currently overlooked in existing RL training frameworks and remains largely unexplored in practice. How can we design reinforcement learning frameworks for LLMs that capture dynamic, user-driven interactions, enable models to adapt in real time to evolve goals and feedback? To investigate the impact of dynamic user roles during the reinforcement learning rollouts, we propose MUA-RL, a novel reinforcement learning framework that, for the first time in the field of agentic tool use, integrates LLM-simulated users into the reinforcement learning loop. MUA-RL is designed to employ end-to-end reinforcement learning to enhance the agent’s ability to iteratively refine its understanding of user intent through ongoing dialogue, while actively invoking tools to fulfill user requests. Unlike approaches that incentivize specific tool-calling formats or success rates at intermediate steps, our framework provides reward solely based on ultimate task completion. This encourages more bold and diverse exploration during reinforcement learning processes, fostering the emergence of more robust and general behavioral patterns. Ultimately, the agent autonomously evolves its capability to handle multi-turn tool-using tasks. Our main contributions are as follows: • We proposed MUA-RL, a novel multi-turn user-interacting reinforcement learning framework that incorporates LLM-simulated users into the reinforcement learning rollouts, and integrated a real, operational database environment for validating the results generated by tool invocation. The framework will be open-sourced on https://github.com/zzwkk/MUA-RL . • To develop the model’s basic capability to invoke tools, we constructed two agentic data synthesis pipelines for high-quality cold-start training: one in which tool responses are simulated by LLMs, and another where tool responses are provided by real MCP server. The high-quality cold-start data will be open-sourced on https://huggingface.co/datasets/zzwkk/MUA-RL-Dataset . • We conducted detailed analysis of the model’s training dynamics throughout the multi-turn user-interacting reinforcement learning process, as well as a comparative examination of its behavior before and after the application of MUA-RL. All MUA-RL models will be open-sourced on https://huggingface.co/zzwkk/MUA-RL-8B , https://huggingface.co/zzwkk/MUA-RL-14B , https://huggingface.co/zzwkk/MUA-RL-32B . 2 Related Work Reinforcement learning. Recent progress in reinforcement learning (RL) for LLMs has moved beyond classical algorithmic frameworks toward more scalable and efficient optimization paradigms. Early work primarily leveraged Proximal Policy Optimization (PPO) [ 6 ] and actor–critic approaches [ 7 ] to align models with human-preferred behaviors. Subsequent developments have introduced policy variants such as RLOO [ 8 ] , GRPO [ 2 ] , and DAPO [ 9 ] , which aim to improve stability and sample efficiency under large-scale training. In parallel, STaR, through its iterative self-derivation process [ 10 ] , and MCTS, via its lookahead-based exploration [ 11 ] , have both demonstrated the ability to significantly reduce reliance on external supervision while substantially improving sample efficiency. More recently, RL has been integrated into broader LLM fine-tuning frameworks, enabling models to adapt flexibly to diverse downstream tasks such as search [ 12 ] , coding [ 13 ] , and multimodal tasks [ 14 ] . These advances underscore the versatility of RL as a general optimization framework for enhancing LLM capabilities, providing a foundation for extending RL techniques into new domains of model alignment and task specialization. Agentic tool use. The development of tool use in LLMs has progressed through a series of methodological stages. Initially, prompting-based approaches relied on carefully designed instructions to elicit tool invocation without additional training [ 15 , 16 ] . While lightweight and flexible, these methods often exhibited unstable tool behavior and struggled with complex tool interactions. Subsequently, supervised fine-tuning (SFT) methods emerged, where models were trained on curated or distilled trajectories to improve their accuracy and consistency in tool use [ 17 , 18 , 19 ] . These approaches achieved more reliable performance but were constrained by the coverage and quality of training data. More recently, reinforcement learning (RL) techniques have been employed to optimize tool-using

Read the complete text