HMCF: A Human-in-the-loop Multi-Robot Collaboration Framework Based on Large Language Models

Zhaoxing Li, Wenbo Wu, Yue Wang, Yanran Xu, William Hunt, Sebastian Stein · 2025

Read HMCF: A Human-in-the-loop Multi-Robot Collaboration Framework Based on Large Language Models by Zhaoxing Li, Wenbo Wu, Yue Wang, Yanran Xu, William Hunt, Sebastian Stein (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-00820.txt.

Introduction

Zhaoxing Li and co-authors published this study in 2025. It frames HMCF, a system where language models coordinate teams of robots with a human on standby. Multi-robot teams struggle with varied machines, new settings, and safety. Each robot here carries its own model agent that knows its skills. The agent turns broad tasks into steps the robot can run. Checks on each plan, plus human supervision, cut the risk of confident nonsense.

In simulations, the framework beat leading task planners, lifting success rates by about five percent. Real-world trials showed it could take on fresh tasks and mixed settings with little human input. People step in only when the system flags a need. The authors pitch the design for jobs like disaster response, where teams must scale fast and stay safe.

Catalog record

CREATORZhaoxing Li, Wenbo Wu, Yue Wang, Yanran Xu, William Hunt, Sebastian Stein
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-00820

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HMCF: A Human-in-the-loop Multi-Robot Collaboration Framework Based on Large Language Models Zhaoxing Li, Wenbo Wu, Yue Wang, Yanran Xu, William Hunt, Sebastian Stein Abstract Rapid advancements in artificial intelligence (AI) have enabled robots to perform complex tasks autonomously with increasing precision. However, multi-robot systems (MRSs) face challenges in generalization, heterogeneity, and safety, especially when scaling to large-scale deployments like disaster response. Traditional approaches often lack generalization, requiring extensive engineering for new tasks and scenarios, and struggle with managing diverse robots. To overcome these limitations, we propose a Human-in-the-loop Multi-Robot Collaboration Framework (HMCF) powered by large language models (LLMs). LLMs enhance adaptability by reasoning over diverse tasks and robot capabilities, while human oversight ensures safety and reliability, intervening only when necessary. Our framework seamlessly integrates human oversight, LLM agents, and heterogeneous robots to optimize task allocation and execution. Each robot is equipped with an LLM agent capable of understanding its capabilities, converting tasks into executable instructions, and reducing hallucinations through task verification and human supervision. Simulation results show that our framework outperforms state-of-the-art task planning methods, achieving higher task success rates with an improvement of 4.76%. Real-world tests demonstrate its robust zero-shot generalization feature and ability to handle diverse tasks and environments with minimal human intervention. I Introduction The rapid progress of artificial intelligence (AI) and robotics has significantly advanced multi-robot systems (MRSs), enabling them to perform increasingly complex tasks with high autonomy and precision [ 1 ] . These systems are deployed in various real-world applications, including disaster response, industrial automation, logistics, and healthcare [ 2 , 3 ] . By coordinating multiple robots, MRSs can efficiently handle large-scale tasks that exceed the capabilities of individual robots. Despite these advancements, several key challenges hinder the effective deployment of MRSs in real-world environments. First, generalization remains a major obstacle—existing methods often rely on task-specific algorithms that require extensive re-engineering when applied to new tasks or environments [ 4 ] . Second, heterogeneity among robots complicates coordination—robots with varying hardware capabilities, sensor configurations, and control mechanisms must seamlessly collaborate while maintaining system-wide efficiency [ 5 ] . Third, existing large language models (LLMs) have shown promise in robot control, but their direct application to MRSs remains underexplored [ 6 ] . While LLMs can interpret high-level commands and generate task plans, they are prone to hallucinations, producing infeasible or unsafe instructions that can compromise real-world deployments [ 7 ] . To address these challenges, we introduce the H uman-in-the-loop M ulti-robot C ollaboration F ramework (HMCF), an innovative LLM-powered framework designed for dynamic and efficient multi-robot collaboration. Our framework combines LLM-based task allocation, real-time human oversight, and decentralized task verification to enable scalable and adaptive multi-robot coordination. In HMCF, each robot operates with an LLM-powered agent that understands its own capabilities, translates tasks into executable instructions, and verifies commands before execution. A central assistant LLM agent manages task assignments, while a human-in-the-loop mechanism provides oversight and intervention when needed. This hybrid approach ensures higher adaptability, safety, and robustness in diverse environments. The key contributions of this paper are as follows: • A novel LLM-powered multi-robot collaboration framework (HMCF) : We develop a human-in-the-loop multi-robot coordination framework that enables efficient and scalable task allocation among heterogeneous robots using LLM-based reasoning. • LLM-assisted generalization across diverse tasks and environments : Unlike traditional task-specific multi-agent collaboration methods, HMCF leverages LLMs to generalize across diverse tasks, reducing the need for extensive task-specific customization. • A user-friendly human-robot interaction interface : We design an intuitive interface that allows users to configure, monitor, and intervene in multi-robot operations using natural language commands, reducing control complexity and operational costs. • LLM hallucination mitigation via human-in-the-loop and task verification : Our framework integrates LLM agents that verify task feasibility before execution and enables human intervention when necessary, ensuring safe and reliable multi-robot collaboration. To evaluate the effectiveness of our framework, we conduct a two-part experimental study. First, we perform a simulated evaluation using the BEHAVIOR-1K benchmark, comparing HMCF against five state-of-the-art multi-robot collaboration methods in diverse environments, which demonstrates a 4.76% improvement in task success rates. Second, we conduct a real-world deployment with heterogeneous robots, validating HMCF’s adaptability and robustness using a team of wheeled and legged robots, showcasing its zero-shot generalization capabilities and seamless human-robot collaboration. The remainder of this paper is organized as follows: Section II reviews related work on LLMs for MRSs and robotics. Section III details the HMCF framework. Section IV presents experiments and evaluations in simulated and real-world environments. Section V discusses experimental findings, limitations, and future directions, and Section VI concludes the paper. II Related Work The integration of Large Language Models (LLMs) into multi-robot systems (MRSs) has shown promise in enhancing collaboration, generalization, and scalability in dynamic environments [ 8 , 9 , 10 ] . LLMs, originally developed for language processing, now facilitate decision-making, task execution, and communication in multi-agent systems (MASs) [ 11 , 12 ] . Their ability to process natural language enables seamless interaction between human users and robots [ 13 , 14 ] . However, key challenges persist in task allocation, execution safety, and hallucination mitigation [ 15 ] . In robot control, LLMs leverage zero-shot and few-shot learning for adaptability in unstructured environments [ 8 , 16 ] . By integrating vision transformers, robots can map visual inputs to actions, improving perception and multi-modal task execution [ 17 , 18 ] . LLMs also enhance high-level task planning, using frameworks like planning domain definition languages for structured decision-making [ 19 , 20 , 21 ] . However, their lack of a deep world model leads to errors in multi-step planning [ 22 , 23 ] . For multi-robot collaboration, LLMs help address communication and task allocation challenges. Systems like SMART-LLM enable natural language-based coordination, improving flexibility in heterogeneous teams [ 24 , 16 , 25 , 26 , 14 , 9 ] . DMRS-2D further decentralizes decision-making through iterative communication [ 27 ] , while Roco integrates LLM-driven collision-free cooperation and human-robot teaming [ 6 ] . However, most approaches only incorporate humans at the task-setting stage, limiting their involvement during execution, where LLM hallucinations can introduce critical errors and risks [ 28 ] . Human-in-the-loop (HITL) systems offer a solution by enabling real-time human oversight and intervention, and ensuring compliance with safety and performance criteria [ 29 ] . HITL integration into LLM-driven MRSs allows natural language command and control, reducing cognitive load and improving task alignment with human expectations [ 9 ] . These mechanisms also support humans as active team members, participating in planning and execution alongside robots [ 27 ] . III Methodology In this section, we first present the overall framework design of HMCF, outlining how tasks are assigned, verified, and dynamically reallocated using two kinds of LLM agents to ensure seamless collaboration. Next, we describe the human-robot interactive interface, which enables intuitive robot management and task verification. We then explain the roles of LLM agents and human users in facilitating robot interactions, reducing hallucinations, and improving task execution efficiency. Finally, we discuss robot deployment cases in both simulation and real-world scenarios. III-A Framework Design Figure 1 illustrates the overall workflow of HMCF, where the system operates sequentially through Step 1: Input

Read the complete text