Transforming Monolithic Foundation Models into Embodied Multi-Agent Architectures for Human-Robot Collaboration
Nan Sun, Bo Mao, Yongchang Li, Chenxu Wang, Di Guo, Huaping Liu · 2025
Read Transforming Monolithic Foundation Models into Embodied Multi-Agent Architectures for Human-Robot Collaboration by Nan Sun, Bo Mao, Yongchang Li, Chenxu Wang, Di Guo, Huaping Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-00797.txt.
Introduction
Nan Sun and five coauthors propose a new shape for robot intelligence in a 2025 paper. Foundation models now unify perception and planning in robotics, promising one model that sees, thinks, and acts. Real service work, though, is spread out, fast changing, and shared among people and robots.
Vision-language models read scenes well but lack a body-aware sense of action and lean on hand-crafted skills. The authors turn a single monolithic model into an embodied multi-agent architecture. Several agents split the cognitive work and coordinate with people. A team design like this may fit messy real workflows better than one giant model.
Catalog record
| CREATOR | Nan Sun, Bo Mao, Yongchang Li, Chenxu Wang, Di Guo, Huaping Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 81 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-00797
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Transforming Monolithic Foundation Models into Embodied Multi-Agent Architectures for Human-Robot Collaboration Nan Sun, Bo Mao, Yongchang Li, Chenxu Wang, Di Guo, Huaping Liu Abstract Foundation models have become central to unifying perception and planning in robotics, yet real-world deployment exposes a mismatch between their monolithic assumption that a single model can handle all cognitive functions and the distributed, dynamic nature of practical service workflows. Vision–language models offer strong semantic understanding but lack embodiment-aware action capabilities while relying on hand-crafted skills. Vision–Language–Action policies enable reactive manipulation but remain brittle across embodiments, weak in geometric grounding, and devoid of proactive collaboration mechanisms. These limitations indicate that scaling a single model alone cannot deliver reliable autonomy for service robots operating in human-populated settings. To address this gap, we present InteractGen, an LLM-powered multi-agent framework that decomposes robot intelligence into specialized agents for continuous perception, dependency-aware planning, decision and verification, failure reflection, and dynamic human delegation, treating foundation models as regulated components within a closed-loop collective. Deployed on a heterogeneous robot team and evaluated in a three-month open-use study, InteractGen improves task success, adaptability, and human–robot collaboration, providing evidence that multi-agent orchestration offers a more feasible path toward socially grounded service autonomy than further scaling standalone models. Fig. 1: A holistic demonstration of how InteractGen coordinates robots and humans in real time. The framework monitors online and offline environments, performs collaborative reasoning across specialized agents, plans composite workflows, and triggers physical execution through heterogeneous robots. Humans are treated as deployable agents—InteractGen clarifies, notifies, and delegates subtasks when appropriate—enabling socially grounded service autonomy. I Introduction Foundation models have rapidly entered the robotics community as a unifying abstraction for perception, language, and action [ 42 , 37 , 17 , 10 ] , yet their real-world impact on service robots remains far from the autonomy required in human-populated environments. Current robotics foundation models fall into two powerful but fundamentally incomplete families. The first comprises embodied Vision-Language-Models (VLM) —such as VeBrain [ 21 ] , RoboBrain [ 14 ] and MiMo-Embodied [ 11 ] —which unify multimodal understanding and spatial reasoning, and often demonstrate strong performance on planning. However, these models typically depend on post-training with benchmark-specific adaptations, in which abilities distilled from larger VLMs are operationalized by generating training data using diverse task templates; however, their action components are generally delegated to hand-designed low-level skill libraries, as in SayCan [ 1 ] , and concrete demonstrations of multi-robot execution, embodiment-aware action, or human-centered deployment remain limited. The second family comprises Vision-Language-Action (VLA) policies , which primarily fine-tune relatively smaller VLMs on robot action datasets using autoregressive language heads [ 5 , 16 ] or diffusion/flow-based action experts [ 4 , 18 , 13 ] , achieving promising reactive manipulation across multiple embodiments and scenes. However, deploying a model in real-world service settings requires comprehensive autonomy : an assistant operating in such environments must simultaneously parse instructions, generate appropriate motion plans, manage uncertainty, and interact fluidly with surrounding human users. Under these demands, current VLAs serve as excellent motor primitives but remain inadequate as assistive systems: they lack persistent task memory, contextual reflection, collaboration awareness, and mechanisms for interacting with humans within realistic workflows. Together, these trends highlight a structural tension: despite substantial isolated capabilities, neither family of foundation models yet realizes the intentional, contextually grounded autonomy required for service robots to deliver meaningful social utility. These considerations point toward a broader insight: the difficulty of deploying current foundation models in real-world service robots does not arise merely from limited scale or training data, but from a deeper mismatch between their monolithic architectures and the distributed, interdependent nature of actual service tasks. Real-world environments inherently involve multiple humans, robots, tools, and information channels, and therefore require cognitive structures capable of arbitrating uncertainty and coordinating heterogeneous embodiments over long horizons. Evidence from digital multi-agent LLM systems such as AutoGen [ 36 ] , MetaGPT [ 12 ] , and AgentVerse [ 7 ] reinforces this view: even without embodiment, reliable long-horizon autonomy typically emerges from role decomposition, reflective loops, tool invocation, and explicit verification steps, rather than from a single large model. When physical robots enter the loop—with real-world dynamics, partial observability, safety constraints, and human unpredictability—the limitations of monolithic foundation models become even more pronounced. For the foreseeable future, it is unlikely that any single VLM or VLA claiming to be an embodied foundation model will simultaneously deliver spatial grounding, cross-embodiment skill composition, multi-agent task allocation, safety arbitration, and socially appropriate interaction. What is needed instead is an architectural shift—one that treats foundation models not as monolithic controllers but as powerful components embedded within structured multi-agent control systems that support collaborative embodied reasoning . Such a design enables the flexibility, interpretability, and competence required for a genuinely general-purpose robotic brain. Motivated by this perspective, we introduce InteractGen, an LLM-powered multi-agent framework enabling collaborative embodied reasoning through specialized agents built on diverse foundation models (Fig. 1 ). InteractGen coordinates heterogeneous robots and humans via five agents for perception, planning, decision making, validation, and reflection. This decomposition extends beyond single-loop paradigms such as ReAct [ 38 ] and Reflexion [ 27 ] , while avoiding decentralized designs requiring a separate LLM per robot as in EMOS [ 6 ] which is hard to scale. InteractGen can also invoke API-level tool calls for GUI operations while serving as a unified “robot brain” for low-level skill libraries or end-to-end VLA policies. By centralizing high-level reasoning and delegating physical execution to generalist controllers or modular primitives, InteractGen treats foundation models as collaborative components rather than monolithic controllers. Crucially, InteractGen is designed as a human-centered interactive system that takes humans as deployable agents capable of taking on delegated subtasks whenever robots encounter safety, authority, or dexterity limits. This allows the system to recognize its own bounds and proactively solicit human assistance when tasks exceed robotic capability in dynamic environments, pushing embodied applications beyond isolated navigation or manipulation toward realistic service workflows that require joint digital–physical coordination. Across controlled evaluations and a three-month open-use deployment, we observed that even simple mobile robots became markedly more useful when embedded in this human-aware loop—acting not merely as executors but as assistants that could anticipate needs, delegate tasks when appropriate, and engage in extended multi-user workflows. These results suggest that multi-agent orchestration, combined with meaningful human involvement, offers a more promising path to socially grounded autonomy than further scaling monolithic foundation models, underscoring that effective human–robot collaboration is central to the future of service robotics. TABLE 1: Comparison between InteractGen and prevailing agentic systems . Integrated denotes single-step action generation, whereas Hierarchical denotes explicit separation between planning and decision making. Validation refers to pre-execution self-checking, and Reflection refers to post-execution reasoning. In the Interaction Skills columns, grey rectangles indicate that the system does not explicitly support this capability in its design or operation. Methods Reasoning Composition Interaction Skills Perception Planning & Decision Validation Reflection Robot Dispatch GUI Operation Human Collaboration ReAct [ 38 ] ✓ Integrated Reflexion [ 27 ] ✓ Integrated ✓ Mobile-Agent-v2 [ 35 ] ✓ Hierarchical ✓ KnowNo [ 26 ] ✓ Integrated ✓ Clarification SMART-LLM [ 15 ] ✓ Integrated CoELA [ 40 ] ✓ Hierarchical ✓ CaPo [ 20 ] ✓ Hierarchical