EMAC+: Embodied Multimodal Agent for Collaborative Planning with VLM+LLM
Shuang Ao, Flora D. Salim, Simon Khan, University of New South Wales, Sydney, Air Force Research Laboratory {shuang.ao, flora.salim}@unsw.edu.au , [email protected] · 2025
Read EMAC+: Embodied Multimodal Agent for Collaborative Planning with VLM+LLM by Shuang Ao, Flora D. Salim, Simon Khan, University of New South Wales, Sydney, Air Force Research Laboratory {shuang.ao, flora.salim}@unsw.edu.au , [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19905.txt.
Introduction
Shuang Ao, Flora Salim, and Simon Khan present a robot agent that plans with both words and images, in a 2025 paper. Language models plan well from text, but robots act in a visual world. Older agents treat the language model as a fixed planner. Its plans can drift from what the robot sees and does.
Their agent, EMAC+, joins a language model with a vision model in a two-way loop. The language model drafts high-level plans. The vision model runs low-level visual tasks and sends back live feedback. That feedback refines the next plan. The agent also learns from its visual exchanges, so its choices improve for the task at hand. The design targets joint planning for embodied work.
Catalog record
| CREATOR | Shuang Ao, Flora D. Salim, Simon Khan, University of New South Wales, Sydney, Air Force Research Laboratory {shuang.ao, flora.salim}@unsw.edu.au , [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19905
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EMAC+: Embodied Multimodal Agent for Collaborative Planning with VLM+LLM Shuang Ao, Flora D. Salim, Simon Khan, University of New South Wales, Sydney, Air Force Research Laboratory {shuang.ao, flora.salim}@unsw.edu.au , [email protected] Abstract Although LLMs demonstrate proficiency in several text-based reasoning and planning tasks, their implementation in robotics control is constrained by significant deficiencies: (1) LLM agents are designed to work mainly with textual inputs rather than visual conditions; (2) Current multimodal agents treat LLMs as static planners, which separates their reasoning from environment dynamics, resulting in actions that do not take domain-specific knowledge into account; and (3) LLMs are not designed to learn from visual interactions, which makes it harder for them to make better policies for specific domains. In this paper, we introduce EMAC+, an Embodied Multimodal Agent that collaboratively integrates LLM and VLM via a bidirectional training paradigm. Unlike existing methods, EMAC+ dynamically refines high-level textual plans generated by an LLM using real-time feedback from a VLM executing low-level visual control tasks. We address critical limitations of previous models by enabling the LLM to internalize visual environment dynamics directly through interactive experience, rather than relying solely on static symbolic mappings. Extensive experimental evaluations on ALFWorld and RT-1 benchmarks demonstrate that EMAC+ achieves superior task performance, robustness against noisy observations, and efficient learning. We also conduct thorough ablation studies and provide detailed analyses of success and failure cases. Code is available at: EMAC+ code. 1 Introduction Recent advancements in Large Language Models (LLMs) have transformed problem-solving in text-centric domains ( Zhu2023MiniGPT4EV ; Shinn2023ReflexionLA ; Shridhar2020ALFWorldAT ; Wang2023DescribeEP ) , facilitating complex planning, reasoning, and tool use via human directives. Nonetheless, implementing LLM-based agents for complex tasks Jiang2023VIMARM in real-world scenarios Huang2023VoxPoserC3 ; Mu2023EmbodiedGPTVP ; Sumers2023DistillingIV remains an open problem. Current multimodal agents typically treat LLMs as static planners, using them solely to generate symbolic instructions executed by predefined visual modules Driess2023PaLMEAE ; Dai2023InstructBLIPTG ; Li2023BLIP2BL . This approach neglects dynamic environmental feedback, hindering adaptability and limiting real-time refinements to action plans. Existing methodologies further exacerbate this disparity by regarding LLMs as static, frozen planners, thus separating their textual reasoning from environment dynamics. Models such as SayCan Ahn2022DoAI or Code-as-Policies Liang2022CodeAP exemplify this limitation, utilizing LLMs to generate symbolic action sequences without accounting for significant obstacles like obscured objects, kinematic constraints, or sensor noise. This static nature prevents LLMs from adapting based on interaction histories or updating their understanding of environment-specific constraints like torque limits or object occlusion patterns. Consequently, LLM-based agents frequently propose impractical actions due to a lack of situational awareness and real-time adaptability. One challenge is that LLM-based agents cannot be directly deployed in a visual world. While LLMs demonstrate impressive abilities in textual interactions, including planning, reflection, and reward shaping, their effectiveness drastically diminishes when directly applied to visual environments. Vision-Language Models (VLMs) have emerged to address this limitation by aligning LLM reasoning with visual inputs. However, existing VLMs predominantly rely on static image-text pair alignments during pretraining, which restricts their ability to understand dynamic visual environments effectively. Even advanced models such as GPT-4V 2023GPT4VisionSC struggle in embodied scenarios like ALFWorld, particularly in zero-shot settings Shridhar2020ALFWorldAT . In such cases, these models typically default to leveraging linguistic priors derived from detected objects rather than adapting plans based on task-specific visual dynamics and interactions. To bridge the gap of deploying LLM in the visual world while maintaining its strengths of both LLM and VLMs, in this paper, we introduce an E mbodied M ulti-modal A gent for C ollaborative planning with LLM + VLM (EMAC+) . EMAC+ allows continuous adaptation of high-level textual action plans based on real-time visual execution feedback. Unlike existing methods, this collaboration empowers the LLM to internalize environment-specific dynamics and improve its domain-specific comprehension, thus generating more accurate and feasible plans for complex robotic tasks. Our approach addresses critical limitations in previous static multimodal methods by providing dynamic, interactive updates to action planning through direct visual feedback. Moreover, with its understanding of physical dynamics, EMAC+ can also generate precise control actions in robotics tasks. While recent VLA models like PaLM-E Driess2023PaLMEAE also output low-level actions, our framework differs in two key ways: (1) A collaborative scheme for LLM + VLM : EMAC+ first uses the prior knowledge from an LLM expert to predict the action sequences. The VLM agent will execute the planned actions to interact with the visual environment, and provide feedback for the LLM expert, including the knowledge of the history trajectories and the environment dynamics. (2) Domain-aware planning : Due to VLM feedback, our LLM expert will possess both contextual knowledge and practical experience of the agent’s execution. In contrast to PaLM-E’s open-loop execution, our LLM systematically refines its plans based on feedback from the VLM. This closed-loop synergy ensures that both models change simultaneously, with the LLM changing to suggest actions that work with the VLM’s control abilities. Figure 1: EMAC+: Embodied Multi-modal Agent for Collaborative Planning with LLM + VLM. EMAC+ takes a task instruction and pixel observations as input to plan the sequence of actions to complete tasks. Moreover, EMAC+ can also be directly deployed into the task/environment, which requires low-level control actions for the interaction. EMAC+ utilizes both the prior knowledge from LLM expert and the domain-specific knowledge from the environment dynamics. We validate EMAC+ in two benchmarks: ALFWorld Shridhar2020ALFWorldAT , which includes numerous tasks in both visual and textual environments, and RT-1 Brohan2022RT1RT , a real-world robotic platform for manipulation. Experiments demonstrate that EMAC+ achieves better performance in both benchmarks with a higher learning efficiency. Meanwhile, EMAC+ demonstrates enhanced generalization capability for out-of-distribution tasks, and exhibits more consistent performance amid environment disturbances. These results highlight three key contributions: (1) a bidirectional training paradigm that harmonizes symbolic reasoning with vision-driven control, enabling the LLM to learn environment dynamics through the VLM’s feedback; (2) an embodied agent that directly generates low-level actions from pixels while retaining the interpretability of language-guided planning; and (3) validation of a co-adaptive LLM-VLM framework in both simulated and real physical environments. Our research propels the boundaries of LLMs as adaptive controllers, facilitating the development of resilient, generalizable agents that acquire knowledge through visual interactions while preserving the natural language-driven reasoning. 2 Related Works A key hurdle for LLM/VLM agents is their inability to generalize beyond training distributions. For example, CLIPort Shridhar2021CLIPortWA demonstrates strong performance on tabletop tasks but fails with novel object configurations or lighting conditions. Similarly, VIMA Jiang2023VIMARM highlights that VLMs trained on static image-text pairs struggle to adapt to dynamic, partially observable RL environments. Recent work by lin2025sim identifies a “sim-to-real perception gap” where agents overfit to simulated visuals and fail under real-world sensory noise. LLM-generated plans (e.g., “pick up the tool”) often lack precision for low-level control, leading to execution failures. Efforts like Distilled Behavior Cloning Liang2024MonitoringAC attempt to mitigate these issues by distilling LLM knowledge into RL policies, but they sacrifice the flexibility of direct language grounding. Due to the nature of language, the textual descriptions of pixel observations are prone to losing important details for robotic control. Unlike these methods, EMAC+ integrates real-time visual feedback into a collaborative loop between LLM and VLM, allowing continuous refinement of plans based on dynamic observations. This interactive process enables EMAC+ to