UAVgent : A Language-Guided Distributed Control Framework
Ziyi Zhang, Xiyu DengGuannan QuYorie NakahiraDepartment of Electrical and Computer EngineeringCarnegie Mellon University5000 Forbes Avenue, Pittsburgh, PA 15213 ziyizhan,xiyud,gqu,[email protected] · 2026
Read UAVgent : A Language-Guided Distributed Control Framework by Ziyi Zhang, Xiyu DengGuannan QuYorie NakahiraDepartment of Electrical and Computer EngineeringCarnegie Mellon University5000 Forbes Avenue, Pittsburgh, PA 15213 ziyizhan,xiyud,gqu,[email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-13212.txt.
Introduction
Multi-drone missions directed by natural language need both flexible task thinking and hard guarantees at the physical layer. Ziyi Zhang and three co-authors propose UAVgent, a three-layer design. A human operator issues language instructions. A language-model supervisor regularly interprets, verifies, and corrects the commanded task against latest state and target guesses. A distributed inner-loop controller tracks the reference using only local relative information.
The authors derive a theory guarantee for tracking scores given bounded disturbances and piecewise-smooth references that jump discretely from language-model updates. The combination shows how centralised language thinking and distributed feedback control can jointly deliver complex behaviours with provable toughness and calm, a template for language-in-the-loop control beyond drones.
Catalog record
| CREATOR | Ziyi Zhang, Xiyu DengGuannan QuYorie NakahiraDepartment of Electrical and Computer EngineeringCarnegie Mellon University5000 Forbes Avenue, Pittsburgh, PA 15213 ziyizhan,xiyud,gqu,[email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-13212
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
UAVgent : A Language-Guided Distributed Control Framework Ziyi Zhang, Xiyu DengGuannan QuYorie NakahiraDepartment of Electrical and Computer EngineeringCarnegie Mellon University5000 Forbes Avenue, Pittsburgh, PA 15213 ziyizhan,xiyud,gqu,[email protected] Abstract We study language-in-the-loop control for multi-drone systems that execute evolving, high-level missions while retaining formal robustness guarantees at the physical layer. We propose a three-layer architecture in which (i) a human operator issues natural-language instructions, (ii) an LLM-based supervisor periodically interprets, verifies, and corrects the commanded task in the context of the latest state and target estimates, and (iii) a distributed inner-loop controller tracks the resulting reference using only local relative information. We derive a theoretical guarantee that characterizes tracking performance under bounded disturbances and piecewise-smooth references with discrete jumps induced by LLM updates. Overall, our results illustrate how centralized language-based task reasoning can be combined with distributed feedback control to achieve complex behaviors with provable robustness and stability. * * footnotetext: These authors contributed equally to this work. 1 Introduction Multi - robot systems are increasingly deployed for surveillance, inspection, environmental monitoring, and emergency response ( 10 ) . Yet specifying and adjusting multi - agent behavior in dynamic environments can be both challenging and tedious, even for an expert user, as it requires not only relevant domain knowledge to determine the optimal action but also rapid adaptation to an evolving environment. The above problem magnifies when the number of drones increases, as designing control policies for each drone and having them collaborate in a time-varying environment requires designing and modifying many parameters that may be prohibitive for any single individual. The development of natural language interfaces offers an approach to lower this barrier by allowing non - experts to express goals and directly map those objectives into low-level actions, such as movement and formation control ( 6 ; 8 ; 13 ) . However, this mapping remains brittle, especially in time-varying environments, because a one-time command might not account for future system changes; and multi - agent coordination amplifies small specification errors into large collective failures ( 14 ; 11 ) . Large language models (LLMs) have recently improved the fidelity with which free - form text can be grounded in structured plans and codes ( 3 ; 4 ) . However, most language - to - robot pipelines are compile - once: a prompt is translated into a plan or policy and then executed until task completion or failure ( 5 ) . This open - loop usage is problematic for operation in a time-varying environment. In such settings, what matters is not only whether the initial plan is reasonable, but also whether the system can continuously incorporate new information to plan for the next action. A human-in-the-loop system offers a natural approach to solving this problem, as a human would be issuing new instructions amid changing objectives or environments. However, in a large multi-agent control system, it may be difficult for a human to pay attention to the entire system all the time. To solve this problem, we propose a mid-layer LLM supervisor as an "assistant" to the human user, which verifies that current execution matches the human’s intent and repair deviations without sacrificing the stability guarantees provided by traditional control. Therefore, the human user would only need to issue a command when a major change occurs, leaving minor adjustments and adaptation to environmental changes to the LLM supervisor. Contribution We introduce a hierarchical, human - in - the - loop control framework in which an LLM-agent supervises execution, translating natural language goals to parameters for each agent, while low-level multi-agent control ensures stability and robustness. Our framework designs a centralized LLM-agent that supervises tasks proposed by a human operator and adapts execution to a changing environment, providing smooth, continuous execution without human intervention. By using an LLM supervisor, we ensure that the commands given to the drones are always aligned with the user’s intention, and by using conventional multi-agent control, we ensure that the drones faithfully adapt in a dynamic environment with minimal human intervention. In addition, we offer a theoretical guarantee that ensures convergence of the control algorithm and provides an upper bound on the controller error between LLM verifications. 2 Related Works Classical Multi-agent Control. Multi-agent control has a long and well-established literature ( 7 ; 9 ) . A common paradigm is to employ a centralized planner that generates intermediate goal representations or control inputs, which are then executed by the team ( 2 ; 1 ) . While central planning can be effective for coordinating complex objectives, it typically depends on a command-and-update structure and frequent communication to disseminate policies to individual agents. This reliance can become a bottleneck in real-time settings where the environment and mission objectives evolve over time. On the other hand, some studies have explored decentralized planning in which each agent’s policy relies solely on its local observations, thereby mitigating the need for real-time communication and providing robustness guarantees ( 16 ; 15 ) . However, those studies often overlook the importance of centralized planning and struggle to achieve complex objectives. Compared to those works, we use an LLM for central planning and periodic auto-correction, but rely on distributed multi-agent control to achieve local robust guarantees. LLM-guided Swarm Control. Recent work has begun to use LLMs as high-level interfaces for specifying and adapting collective behaviors in robot swarms. A first line of research treats language as a convenient task specification layer and learns a low-level multi-robot policy that realizes the requested pattern or motion. For instance, 6 studies language-guided pattern formation using multi-agent reinforcement learning, where natural language prompts describe target spatial arrangements that are then achieved by decentralized robot policies. Moreover, 8 explores linguistic orchestration of UAV flocking, using an LLM to translate descriptive commands (e.g., formation type and motion style) into flock-level control objectives. Going beyond a single centralized LLM, 12 propose LLM2Swarm , in which LLM components support responsive reasoning, planning, and collaboration for swarms, highlighting the potential of multi-agent LLM architectures to mediate coordination. Zero-shot approaches have also emerged: 13 demonstrates context-aware pattern formation from high-level user intent without task-specific fine-tuning, including setups where formations may be specified via richer modalities. Despite rapid progress, most existing language-to-swarm pipelines remain largely ”compile-once” at the high level: a language module produces a formation/plan, which is then executed with limited formal linkage between language-level edits and closed-loop stability. This gap is particularly acute in dynamic multi-target scenarios, where ambiguous instructions, drifting execution, or transient disturbances can cause cascading coordination errors. Our work is positioned to address this gap by coupling language-in-the-loop supervision (periodic intent verification and constrained goal edits) with a robust distributed controller that provides explicit stability/robustness guarantees between LLM updates. 3 Algorithm Design We propose the framework UAVgent , a three-layer hierarchy that couples (i) natural-language commands, (ii) LLM-based runtime verification/correction, and (iii) a distributed inner-loop controller with robustness guarantees. The layers operate on separated time scales: the user issues sporadic commands (outer layer), the LLM performs K periodic supervision at times \{t_{k}\}_{k\in\{0,\dots,K-1\}} with t_{k}-t_{k-1}=:\Delta_{t} for all k\in\{1,\dots,K-1\} (middle layer), and the multi-agent controller runs continuously (inner layer). In the environment, a human user commands N_{a} drones in \mathbb{R}^{d} to achieve objectives that may involve N_{b} independent