How Good are Foundation Models in Step-by-Step Embodied Reasoning?
Dinura Dissanayake 1 , Ahmed Heakl 1 , Omkar Thawakar 1 , Noor Ahsan 1 , Ritesh Thawkar 1 , Ketan More 1 , Jean Lahoud 1 , Rao Anwer 1 , Hisham Cholakkal 1 , Ivan Laptev 1 , Fahad Khan 1, 2 , Salman Khan 1, 3 1 Mohamed bin Zayed University of AI, 2 Linköping University, 3 Australian National Univers · 2025
Read How Good are Foundation Models in Step-by-Step Embodied Reasoning? by Dinura Dissanayake 1 , Ahmed Heakl 1 , Omkar Thawakar 1 , Noor Ahsan 1 , Ritesh Thawkar 1 , Ketan More 1 , Jean Lahoud 1 , Rao Anwer 1 , Hisham Cholakkal 1 , Ivan Laptev 1 , Fahad Khan 1, 2 , Salman Khan 1, 3 1 Mohamed bin Zayed University of AI, 2 Linköping University, 3 Australian National Univers (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-15293.txt.
Introduction
Dinura Dissanayake and eleven co-authors benchmark how well foundation models reason step by step in embodied settings in a 2025 AI paper. Robots and agents in the physical world must act effectively, safely, and in ways that respect spatial context. Large multimodal models understand images and language impressively, but structured reasoning for physical tasks is a different demand.
Their benchmark, FoMER, scores models across varied scenarios requiring them to read multimodal observations, weigh physical constraints and safety, and produce valid next actions in natural language. Step-by-step performance is the explicit target, since a correct final answer reached by broken reasoning fails in the real world. The study maps where today's foundation models reason soundly about bodies and spaces, and where their chains of thought snap.
Catalog record
| CREATOR | Dinura Dissanayake 1 , Ahmed Heakl 1 , Omkar Thawakar 1 , Noor Ahsan 1 , Ritesh Thawkar 1 , Ketan More 1 , Jean Lahoud 1 , Rao Anwer 1 , Hisham Cholakkal 1 , Ivan Laptev 1 , Fahad Khan 1, 2 , Salman Khan 1, 3 1 Mohamed bin Zayed University of AI, 2 Linköping University, 3 Australian National Univers |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-15293
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
How Good are Foundation Models in Step-by-Step Embodied Reasoning? Dinura Dissanayake 1 , Ahmed Heakl 1 , Omkar Thawakar 1 , Noor Ahsan 1 , Ritesh Thawkar 1 , Ketan More 1 , Jean Lahoud 1 , Rao Anwer 1 , Hisham Cholakkal 1 , Ivan Laptev 1 , Fahad Khan 1, 2 , Salman Khan 1, 3 1 Mohamed bin Zayed University of AI, 2 Linköping University, 3 Australian National University Abstract Embodied agents operating in the physical world must make decisions that are not only effective but also safe, spatially coherent, and grounded in context. While recent advances in large multimodal models (LMMs) have shown promising capabilities in visual understanding and language generation, their ability to perform structured reasoning for real-world embodied tasks remains underexplored. In this work, we aim to understand how well foundation models can perform step-by-step reasoning in embodied environments. To this end, we propose the Foundation Model Embodied Reasoning (FoMER) benchmark, designed to evaluate the reasoning capabilities of LMMs in complex embodied decision-making scenarios. Our benchmark spans a diverse set of tasks that require agents to interpret multimodal observations, reason about physical constraints and safety, and generate valid next actions in natural language. We present (i) a large-scale, curated suite of embodied reasoning tasks, (ii) a novel evaluation framework that disentangles perceptual grounding from action reasoning, and (iii) empirical analysis of several leading LMMs under this setting. Our benchmark includes over 1.1k samples with detailed step-by-step reasoning across 10 tasks and 8 embodiments, covering three different robot types. Our results highlight both the potential and current limitations of LMMs in embodied reasoning, pointing towards key challenges and opportunities for future research in robot intelligence. Project page: https://mbzuai-oryx.github.io/FoMER-Bench/ 1 Introduction Embodied agents, including robots and virtual agents that perceive and act within the physical world, are commonly deployed in complex, unstructured environments. Whether navigating cluttered homes, manipulating objects in a kitchen, or assisting humans in dynamic settings, these agents must target effective decision-making. To achieve an intelligent level of decision-making, the embodied agents require reasoning: the ability to understand task goals, assess spatial relationships among objects and surfaces, and account for safety and physical feasibility before taking an action. Figure 1 : Example illustrating the final answer and step-by-step reasoning from Gemini ( ) comanici2025gemini25 and Qwen ( ) Qwen2.5-VL for a given video and text prompt. While both models correctly identify the action as “withdraw bolt,” the reasoning they provide differs significantly. This underscores the importance of evaluating the final answer as well as the underlying reasoning. Table 1 : A comparison of various Embodied and Physical AI benchmarks. We summarize key features across benchmarks, including input modalities, question formats, presence of step-by-step reasoning trails, number of annotated questions, annotation methods, diversity of tasks and embodiments, and the types of robots involved. Our benchmark (last row) is distinguished by explicitly incorporating reasoning traces, supporting a variety of question types, and covering a broader set of tasks and robotic platforms compared to prior work. Benchmark Modality Question Reasoning # of Annotation # of # of Embodiments # of Robot Types Trails Questions Tasks Videos Types Cosmos-R1 nvidia2025cosmosreason1physicalcommonsense Videos + Text MCQ, TF ✗ 510 Automatic + Human 5 501 Agibot-G1, Widow-X, UR5e, Human Demonstrations 2 Robo2VLM-1 chen2025robo2vlmvisualquestionanswering Images + Text MCQ ✗ 6676 Automatic 5 463 Franka, UR5,fanuc-mate-200id, xArm, Kuka-LBR-IIWA 1 HRIBench shi2025hribenchbenchmarkingvisionlanguagemodels Image + Text Open ✗ 1000 - 5 - Quori, Kuri, Franka Emika 3 OpenEQA OpenEQA2023 Videos + Text Open ✗ 1600 Human Generated 2 180 PBench PBench_NVIDIA Image + Text TF ✗ 5636 Automatic + Human 4 1044 - - ECEBench dang2025ecbenchmultimodalfoundationmodels Video + Text MCQ + Open ✗ Human Generated 3 378 - - Ours Videos/Image Frames + Text TF + MCQ + Open ✓ 1112 Automatic + Human 10 758 Agibot G1, Widow X, UR5e, Human Demonstrations, Jackal, Hello stretch, Franka 3 While classical robotics addresses planning and control through explicit models of geometry and physics, the emergence of large multimodal models (LMMs) opens a new direction. These models, which integrate vision and language through large-scale pretraining, have demonstrated impressive capabilities in visual understanding and language generation. Prior work has applied LMMs or vision-language models to embodied contexts rt22023arxiv ; kim24openvla . However, their potential for supporting complex decision-making, especially based on reasoning, remains largely underexplored. On the other hand, outside of the field of embodied agents, several approaches have been presented to study and benchmark reasoning in LMMs thawakar-etal-2025-llamav . These approaches often target general reasoning, such as commonsense, temporal, and abstract logic, among others. Nevertheless, these works are typically detached from the embodied setting and fail to capture reasoning types that are crucial for physical interaction, such as spatial alignment, object affordance, and safety awareness. As a result, there is a lack of clear understanding of whether and how LMMs can reason effectively in the context of embodied tasks. This gap is particularly evident when examining how LLMs generate their final answers. Figure 1 shows how a model can arrive at the right conclusion through faulty reasoning. In this example, both Gemini 2.5 Pro comanici2025gemini25 and Qwen 2.5-VL Qwen2.5-VL reach the same final answer but follow completely different reasoning paths. This highlights the need to evaluate not only the final answers, but also the reasoning trails behind them. In this work, we introduce the Foundation Model Embodied Reasoning benchmark (FoMER), specifically designed to evaluate embodied reasoning in LLMs. Our benchmark comprises a diverse set of scenarios in which an agent must respond to prompts in natural language, grounded in both visual observations and a task description. We carefully annotate each scenario with a valid action response, alongside fine-grained reasoning labels that cover spatial relations, safety constraints, and task alignment. When compared to existing embodied reasoning benchmarks (Table 1 ), our benchmark offers the largest coverage, both in the question diversity and in reasoning diversity. To assess a model’s performance, we propose an evaluation framework that measures not only the accuracy of the selected action but also the quality of the underlying reasoning, as judged against human-annotated reasoning. This enables us to check cases where a model chooses responses based on guesses, rather than truly understanding the underlying decision logic. In summary, our contributions are as follows: • We present a large-scale benchmark for evaluating embodied reasoning in LMMs, spanning more than 1k scenarios with detailed reasoning annotations. • We propose an evaluation framework that emphasizes grounded reasoning, introducing new metrics that capture both action validity and reasoning correctness. • We benchmark several state-of-the-art LMMs under this framework, highlighting strengths and limitations in their ability to reason about real-world embodied tasks. 2 Related Work While Foundation Models have achieved remarkable fluency, establishing them with reasoning abilities, especially in spatial contexts, remains an actively researched challenge. Reasoning in Foundation Models. Reasoning is a core component of intelligence, which is supported by the ability to formulate decisions, draw conclusions, and generalize conclusions in different domains. Recent Large Language Models (LLMs) and Large Multimodal Models (LMMs) have demonstrated remarkable improvements in factual recall and language generation openai2024gpt4technicalreport ; chen2024internvl ; bai2023qwenvlversatilevisionlanguagemodel . Still, their performance in complex questions requiring multi-step reasoning and physical common sense falls short compared to human cognition.