EmbodiedEval : Evaluate Multimodal LLMs as Embodied Agents
Zhili Cheng Yuge Tu Ran Li Shiqi Dai Jinyi Hu Shengding Hu, Jiahao Li Yang Shi Tianyu Yu Weize Chen Lei Shi Maosong Sun [0.5em] Tsinghua University {chengzl22, hu-jy21}@mails.tsinghua.edu.cn · 2025
Read EmbodiedEval : Evaluate Multimodal LLMs as Embodied Agents by Zhili Cheng Yuge Tu Ran Li Shiqi Dai Jinyi Hu Shengding Hu, Jiahao Li Yang Shi Tianyu Yu Weize Chen Lei Shi Maosong Sun [0.5em] Tsinghua University {chengzl22, hu-jy21}@mails.tsinghua.edu.cn (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-11858.txt.
Introduction
A model that chats about pictures may still fumble at acting in a world. Zhili Cheng and co-authors build EmbodiedEval in 2025, an interactive benchmark that grades multimodal language models as embodied agents inside simulated 3D scenes. Its 328 tasks span 125 scenes across navigation, object handling, social interaction, and questions about attributes and space.
Top current models fell far short of human performance. The failures map where embodied skill lags: acting over time, not just describing a frame. The team open-sources the data and simulator so developers can train against the gaps instead of guessing at them.
Catalog record
| CREATOR | Zhili Cheng Yuge Tu Ran Li Shiqi Dai Jinyi Hu Shengding Hu, Jiahao Li Yang Shi Tianyu Yu Weize Chen Lei Shi Maosong Sun [0.5em] Tsinghua University {chengzl22, hu-jy21}@mails.tsinghua.edu.cn |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-11858
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EmbodiedEval : Evaluate Multimodal LLMs as Embodied Agents Zhili Cheng Yuge Tu Ran Li Shiqi Dai Jinyi Hu Shengding Hu, Jiahao Li Yang Shi Tianyu Yu Weize Chen Lei Shi Maosong Sun [0.5em] Tsinghua University {chengzl22, hu-jy21}@mails.tsinghua.edu.cn Abstract Multimodal Large Language Models (MLLMs) have shown significant advancements, providing a promising future for embodied agents. Existing benchmarks for evaluating MLLMs primarily utilize static images or videos, limiting assessments to non-interactive scenarios. Meanwhile, existing embodied benchmarks are task-specific and not diverse enough, which do not adequately evaluate the embodied capabilities of MLLMs. To address this, we propose EmbodiedEval , a challenging and comprehensive benchmark to evaluate MLLMs’ interactive capabilities in embodied tasks within a unified simulation and evaluation framework tailored for MLLMs. We evaluate the state-of-the-art MLLMs on EmbodiedEval and find that they have a significant shortfall compared to human level on embodied tasks. Our analysis demonstrates the limitations of existing MLLMs in embodied capabilities, providing insights for their future development. Figure 1: Examples of the five task categories and performance overview of EmbodiedEval . The embodied agent powered by MLLMs is required to finish the given task in a 3D simulation environment. 1 Introduction In recent years, Multimodal Large Language Models (MLLMs) ( OpenAI 2023 ; Team et al. 2023 ; Liu et al. 2024a ) have demonstrated strong capabilities in understanding and reasoning across vision and language tasks. With the rapid development of MLLMs, a rich set of benchmarks ( Yue et al. 2024 ; Liu et al. 2023b ; Fu et al. 2023 ; Li et al. 2023a ) has been developed. Beyond these basic tasks that focus on non-interactive visual scenes, researchers are actively trying to expand MLLMs as embodied agents in interactive environments, which require the model to interpret multimodal inputs into actions ( Ahn et al. 2022 ; Driess et al. 2023 ; Mu et al. 2024 ) . To accomplish this, MLLMs are expected to integrate a multitude of capabilities that enable them to interact effectively with the environment, including ego-centric perception ( Cheng et al. 2024a ) , visual grounding ( Anderson et al. 2018b ; Zhang et al. 2024b ) , spatial reasoning ( Chen et al. 2024 ) and episodic memory ( Majumdar et al. 2024 ) . Benchmark Scene. Task. Disc. Ego. Nav. Obj. So. Ans. MME ( Fu et al. 2023 ) - ✓ ✓ ✗ ✗ ✗ ✗ ✓ EgoPlan etc. ( Chen et al. 2023b ; Cheng et al. 2023 ) - ✓ ✓ ✓ ✗ ✗ ✗ ✓ OpenEQA ( Majumdar et al. 2024 ) - ✓ ✓ ✓ ✗ ✗ ✗ ✓ EQA etc. ( Das et al. 2018 ; Yu et al. 2019 ; Tan et al. 2023 ) ✗ ✗ ✓ ✓ ✓ ✗ ✗ ✓ ALFRED ( Shridhar et al. 2020 ) ✗ ✗ ✗ ✓ ✓ ✓ ✗ ✗ BEHAVIOR ( Srivastava et al. 2022 ) ✗ ✓ ✗ ✓ ✓ ✓ ✗ ✗ EQA-MX ( Islam et al. 2024 ) ✗ ✗ ✓ ✓ ✗ ✗ ✓ ✓ EmbodiedEval ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Table 1: Comparison of EmbodiedEval with previous benchmarks. The abbreviations in the table headers, from left to right, represent: Scene diversity (beyond household scenes), Task diversity (beyond task templates), Disc rete action space (for MLLMs evaluation), Ego centric vision, Nav igation involved, Obj ect interaction involved, So cial interaction involved, and Ans wering questions involved. However, the comprehensive evaluation of MLLMs in embodied tasks remains largely unexplored. First, existing benchmarks for embodied tasks lack diversity in both tasks and scenes. For instance, ALFRED ( Shridhar et al. 2020 ) includes just seven predefined tasks (e.g., "pick and place") within four room types. Second, several benchmarks Anderson et al. 2018b ; Qi et al. 2020a impose rigid input-output formats, e.g. 3D points, making it inefficient or even infeasible to evaluate mainstream MLLMs. Third, certain benchmarks ( Li et al. 2024 ; Liu et al. 2023a ; Jia et al. 2024 ) try to evaluate LLMs’ embodied performance by representing environments with textual descriptions, relying heavily on text-based states. This downplays critical embodied skills such as visual grounding and spatial reasoning, which are essential for real-world interaction. To address this gap, we introduce the first comprehensive benchmark for evaluating MLLMs’ embodied capabilities in interactive environments. The key features of EmbodiedEval are as follows: (I) Diverse Interactions. EmbodiedEval provides a simulation framework that supports a wide range of interactions with objects and humans in realistic 3D environments. Agents need to interact with the environment to gather information or alter its state to complete tasks. The ego-centric visual information will serve as input to the MLLMs to make the decision. (II) Diverse Tasks. Unlike previous work that relied on predefined task templates, our tasks are systematically generated and carefully selected to ensure both high quality and diversity. EmbodiedEval includes novel tasks that assess a broad spectrum of abilities, enabling a more comprehensive evaluation of the model’s capabilities. (III) Diverse Scenes. Our scenes offer significant diversity in terms of objects and spaces, encompassing house rooms, large residences, and public areas such as gyms, stores, and offices. This variety helps minimize the impact of specific scene types on the model’s generalization. Experiment results on EmbodiedEval reveal that mainstream MLLMs largely fall short of human-level performance on embodied tasks. Model performance varies widely across different task categories, with a notable drop in spatial and long-horizon tasks. EmbodiedEval provides insights for further improvements in MLLMs’s capability in grounding, spatial reasoning, planning, and exploration. 2 Related Works Multimodal Large Language Models. By connecting vision modules with LLMs, LLaVA ( Liu et al. 2024a ) pioneers research in MLLMs through visual instruction tuning. Many work further improves the MLLMs from various aspects, including detailed captioning ( Chen et al. 2023a ) , trustworthy response ( Yu et al. 2024a ; Yu et al. 2024b ) , multilingual capabilities ( Hu et al. 2023 ) , visual grounding ( Peng et al. 2023 ; You et al. 2023 ) and video understanding ( Lin et al. 2023 ; Liu et al. 2024c ) . Evaluation for MLLMs. Mainstream benchmarks for MLLMs mainly focus on perception and cognitive evaluation ( Fu et al. 2023 ; Liu et al. 2023b ; Yue et al. 2024 ; Fu et al. 2024 ) and some benchmarks focus on more challenging tasks ( Lu et al. 2024 ; He et al. 2024 ; Singh et al. 2019 ; Liu et al. 2023c ; Yang et al. 2024a ; Yue et al. 2024 ) . Additionally, certain benchmarks ( Fan 2019 ; Chen et al. 2023b ; Cheng et al. 2023 ; Majumdar et al. 2024 ; Szot et al. 2023 ) have been designed to evaluate the egocentric capabilities of MLLMs using egocentric images or videos. However, these benchmarks use static question-answering pairs without interacting with environments. Figure 2: The evaluation process of EmbodiedEval . The task description and ego-centric observation history will be input for the model. The environment will respond to the action from the model output with a new observation. Benchmarks for Embodied Agents. Existing benchmarks or datasets for embodied agents cover several areas such as embodied