Evaluation of Habitat Robotics using Large Language Models

William Li, Lei Hamilton, Kaise Al-natour, Sanjeev Mohindra · 2025

Read Evaluation of Habitat Robotics using Large Language Models by William Li, Lei Hamilton, Kaise Al-natour, Sanjeev Mohindra (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06157.txt.

Introduction

William Li and co-authors graded language models on cooperative robot tasks in simulated kitchens, in this 2025 paper. Two robot agents must work together on household tasks in varied room layouts. The benchmark varies who plans, who sees what, and how much each agent knows.

Several frontier models were tested. A reasoning model beat non-reasoning rivals across all setups: central or split planning, full or partial views. The gap held in each configuration tried. The authors read the result as a hopeful sign for language-model control of embodied robots, and as a call to test reasoning skill in such settings.

Catalog record

CREATORWilliam Li, Lei Hamilton, Kaise Al-natour, Sanjeev Mohindra
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-06157

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluation of Habitat Robotics using Large Language Models William Li, Lei Hamilton, Kaise Al-natour, Sanjeev Mohindra Abstract This paper focuses on evaluating the effectiveness of Large Language Models at solving embodied robotic tasks using the Meta PARTNER benchmark. Meta PARTNR provides simplified environments and robotic interactions within randomized indoor kitchen scenes. Each randomized kitchen scene is given a task where two robotic agents cooperatively work together to solve the task. We evaluated multiple frontier models on Meta PARTNER environments. Our results indicate that reasoning models like OpenAI o3-mini outperform non-reasoning models like OpenAI GPT-4o and Llama 3 when operating in PARTNR’s robotic embodied environments. o3-mini displayed outperform across centralized, decentralized, full observability, and partial observability configurations. This provides a promising avenue of research for embodied robotic development. I Background In 2020, OpenAI released GPT-3 [ 1 ] which ignited considerable interest from the research community regarding Large Language Models (LLM). In particular, robotics researchers began looking toward LLMs as a way of integrating intelligence into their automata. Prior to the popularity of LLMs, most researchers were focused on designing robotic controls or leveraging reinforcement learning [ 2 ] to perform robotic manipulation. With the release of ever more powerful LLMs, robotics researchers began looking at ways to incorporate LLMs into their development. In 2023, a global consortium of researchers from industry and academia released the Open-X [ 3 ] dataset for robotic research. The Open-X dataset provided a dataset which could be used to train robotic manipulation with corresponding images and natural language instructions. This paved the way for developing Vision Language Action (VLA) models which integrated language, vision, and robotic control. In 2024, Berkley University released Octo [ 4 ] ; a seminal paper on using VLA Transformer to perform robotic manipulation. In that same year OpenVLA [ 5 ] was released, demonstrating improved VLA performance built on top of Llama2 [ 6 ] . Diffusion methods such as RDT-1B [ 7 ] further improved upon OpenVLA to demonstrate additional performance gains under certain situations. In parallel to robotic research, the number of simulators supporting robotic research was growing as well. Popular robotic simulators have emerged such as Maniskill [ 2 ] , Nvidia Isaac Sim [ 8 ] , AI2Thor [ 9 ] , Robocasa [ 10 ] and Genesis [ 11 ] . Many of these simulators were designed to support robotic manipulation by rendering physics accurately. In 2024, Meta released a benchmark designed specifically to study embodied robotics in collaborative environments with explicit LLM support. The benchmark was called Planning and Reasoning in Embodied Multi-agent Tasks (PARTNR) [ 12 ] ; it was designed to study robot coordination using Large Language Models (LLMs) for indoor scenes. PARTNR was built on the Habitat [ 13 ] simulator and was specifically designed to benchmark heterogeneous multiagent coordination using LLMs: PARTNR tasks exhibit characteristics of everyday tasks, such as spatial, temporal, and heterogeneous agent capability constraints. We employ a semi-automated task generation pipeline using Large Language Models (LLMs), incorporating simulation in the loop for grounding and verification. PARTNR stands as the largest benchmark of its kind, comprising 100,000 natural language tasks, spanning 60 houses and 5,819 unique objects. PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks PARTNR provides a standardized baseline for evaluating planning , coordination , and perception for multiagent embodied robots in simulated indoor settings. The tasks require planning because agents must apply the correct sequence of actions in order to accomplish their tasks. PARTNR simulates multiagent collaborative tasks which require coordination . Environments can be setup as either fully observable or partially observable to test the perception capabilities of LLMs. Tasks and environments are procedurally generated ensuring a diverse range of unique tests. Despite state-of-the art performance of LLMs in advance math [ 14 ] , graduate level questions [ 15 ] , and coding capabilities [ 16 , 17 , 18 , 19 ] ; LLMs lack an embodied understanding of the physical world. This gap between LLM’s understanding of the physical world prevents generalized deployment of robotics into the real world. In the PARTNR [ 12 ] paper they stated that state-of-the art LLM models displayed poor performance in embodied tasks: The analysis reveals significant limitations in SoTA models, such as poor coordination and failures in task tracking and recovery from errors…Overall, PARTNR highlights significant challenges facing collaborative embodied agents and aims to drive research in this direction… PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks Since the release of the original PARTNR benchmark, multiple new frontier models have been released such as OpenAI GPT-4o, OpenAI o3-mini, DeepSeek R1, etc. OpenAI GPT-4o is considered a non-reasoning model whereas DeepSeek R1 and OpenAI o3-mini are considered reasoning models. The purpose of our study was to determine how well these models perform on the PARTNR benchmark. Chain of Thought (CoT), used in reasoning models, has shown dramatically improved performance over non-reasoning models using test time compute. However, they utilize significantly more tokens and often take longer to produce an answer. We wanted to understand whether reasoning models that utilized CoT could help with embodied tasks in robotics. II Simulation PARNTR provides a common framework for embodied robotic agents to interact within a digital indoor environment to solve a task. Two robotic agents are simulated within the environment: (1) a humanoid robot and (2) a Boston Dynamics Spot robot. It should be noted that the spot robot and the humanoid robot have different capabilities. The humanoid robot can perform all the actions of the Spot robot, but has additional capabilities. Spot Robot Capabilities: • Close: Used for closing an articulated entity. You must provide the name of the furniture you want to close. • Explore: Search a specific room by visiting various receptacles or furniture in that room. The input to the skill is the EXACT name of the room to be visited. Use the room names provided in the house description. This tool exhaustively explores the specified room. • Navigate: Used for navigating to an entity. You must provide the name of the entity you want to navigate to. • Open: Used for opening an articulated entity. You must provide the name of the furniture you want to open. • Pick: Used for picking up an object. You must provide the name of the object to be picked. The agent cannot hold more than one object at a time. • Place: Used for placing an object on a target location. You need to provide the name of the object to be placed, the name of the furniture where it should be placed, spatial relation (”on” or ”within”) describing the relation between the object and furniture. The object to be placed must already be held by the agent (i.e. picked previously). • Rearrange: Used for moving an object from its current location to the target location. You need to provide the name of the object to be moved, the name of the furniture where it should be moved, spatial relation (”on” or ”within”) describing the relation between the object and furniture. This will automatically pick the specified object and move to the target furniture and attempt to place it. • Wait:

Read the complete text