HEAL: An Empirical Study on 𝑯 ​ allucinations \bm{H}\text{allucinations} in 𝑬 \bm{E} mbodied 𝑨 \bm{A} gents Driven by 𝑳 \bm{L} arge Language Models

Trishna Chakraborty, Udita Ghosh, Xiaopan Zhang, Fahim Faisal Niloy, Jiachen Li, Amit K. Roy-Chowdhury, Chengyu Song, [email protected] · 2025

Read HEAL: An Empirical Study on 𝑯 ​ allucinations \bm{H}\text{allucinations} in 𝑬 \bm{E} mbodied 𝑨 \bm{A} gents Driven by 𝑳 \bm{L} arge Language Models by Trishna Chakraborty, Udita Ghosh, Xiaopan Zhang, Fahim Faisal Niloy, Jiachen Li, Amit K. Roy-Chowdhury, Chengyu Song, [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15065.txt.

Introduction

Trishna Chakraborty and co-authors present a 2025 arXiv study of hallucinations in robot agents driven by language models. Such agents may hunt for rooms or objects that do not exist when a task clashes with the scene. The team built a probe set that manufactures these clashes, raising hallucination rates up to forty times over plain prompts.

Twelve models tried tasks in two simulated worlds. Models reasoned fluently yet failed to call out impossible tasks, wandering on false premises. The authors chart which inconsistencies trigger the failures and sketch what ideal behavior looks like in each case, as guidance for sturdier planners.

Catalog record

CREATORTrishna Chakraborty, Udita Ghosh, Xiaopan Zhang, Fahim Faisal Niloy, Jiachen Li, Amit K. Roy-Chowdhury, Chengyu Song, [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15065

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HEAL: An Empirical Study on 𝑯 ​ allucinations \bm{H}\text{allucinations} in 𝑬 \bm{E} mbodied 𝑨 \bm{A} gents Driven by 𝑳 \bm{L} arge Language Models Trishna Chakraborty, Udita Ghosh, Xiaopan Zhang, Fahim Faisal Niloy, Jiachen Li, Amit K. Roy-Chowdhury, Chengyu Song, [email protected] Abstract Large language models (LLMs) are increasingly being adopted as the cognitive core of embodied agents. However, inherited hallucinations, which stem from failures to ground user instructions in the observed physical environment, can lead to navigation errors, such as searching for a refrigerator that does not exist. In this paper, we present the first systematic study of hallucinations in LLM-based embodied agents performing long-horizon tasks under scene–task inconsistencies. Our goal is to understand to what extent hallucinations occur, what types of inconsistencies trigger them, and how current models respond. To achieve these goals, we construct a hallucination probing set by building on an existing benchmark, capable of inducing hallucination rates up to 40 Ă— 40\times higher than base prompts. Evaluating 12 models across two simulation environments, we find that while models exhibit reasoning, they fail to resolve scene-task inconsistencies—highlighting fundamental limitations in handling infeasible tasks. We also provide actionable insights on ideal model behavior for each scenario, offering guidance for developing more robust and reliable planning strategies. 1 Introduction Recent advances in the reasoning and generalization capabilities of large language models (LLMs) Chang et al. 2024 ; Wei et al. 2022 have led to their increasing adoption as the cognitive core Mai et al. 2023 of embodied agents Zhang et al. 2024b ; Kannan et al. 2024 ; Dorbala et al. 2023 , enabling these systems to interpret instructions in natural language and formulate action plans in complex environments. However, LLMs have well-known vulnerabilities Liu et al. 2024 ; Chakraborty et al. 2024 . Consequently, LLM-driven agents Xiang et al. 2023 ; Yang et al. 2025 inherit not only the vast world knowledge and reasoning capability of LLMs, but also their limitations Jiao et al. 2024 ; Zhang et al. 2025 ; most notably a persistent tendency to hallucinate Perković et al. 2024 ; Sriramanan et al. 2024 . Figure 1: Object hallucination rates ( C_{O} ) on our hallucination probing set in VirtualHome. Higher values indicate more hallucination, with Scene Task Contradiction triggering the highest rates in nearly all models. Figure 2: Overview of our settings. (A) The base pipeline in the existing benchmark Li et al. 2024b begins with a scene parser that extracts structured textual scene information from raw visual input. Combined with the natural language task description, this is processed by an LLM to generate symbolic goals in Linear Temporal Logic (LTL) (see subsection 3.1 ). (B) Examples of hallucinations. Output elements highlighted in red indicate hallucinated content that is not grounded in the scene information, i.e., inconsistent with the observed environment. These examples demonstrate that when inconsistencies arise between the scene information and the given task description, the LLM fails to reconcile the two and generates incorrect plans or object references. Given the base prompt, we systematically modify two core input components—the task description and scene information—to elicit hallucinations. Our four controlled modifications of the base prompts are: under task description variation, (i) Distractor Injection—adds non-existent objects to the task description; and under scene variation, (ii) Task Relevant Object Removal—omits key objects from the scene; (iii) Synonymous Object Substitution—replaces scene objects with synonyms; and (iv) Scene Task Contradiction—introduces conflicts between the task and scene. While hallucination is a well-recognized limitation of LLMs Tonmoy et al. 2024 ; Rawte et al. 2023a , its manifestation in embodied agents is qualitatively different. Unlike conversational systems, where hallucinations often result in factual errors or incoherent replies Zhou et al. 2023 ; Yu et al. 2024a , hallucinations in embodied agents stem from a failure to ground user-provided task instructions in the observed physical environment. This misalignment can lead to consequences far more serious than a simple textual error. For example, if a robot is instructed to “put the knives in the dishwasher” but no dishwasher is present, an LLM unable to reconcile the task with the observed scene may hallucinate the existence of the dishwasher , and include it in the generated plan to follow the user instruction. This can cause the robot to place sharp utensils in an empty cabinet or try to press buttons on a bare wall, leading to physical damage, safety hazards, and wasted battery. Such behaviors highlight the need for scene-task-consistent planning in LLM-based agents. Motivated by these limitations of current LLMs—most of which are optimized to complete tasks under ideal conditions—we shift our focus to understand their failure modes. Specifically, we aim to answer the following research questions: RQ1: To what extent do LLM-based embodied agents hallucinate under scene–task inconsistencies; what types of mismatches are more likely to trigger hallucinations; and what limitations do these failures reveal? RQ2: Does the absence of hallucination imply correct planning? What are the ideal behaviors in these scenarios for more robust planning? Although prior works explore LLMs in embodied agents Majumdar et al. 2024 ; Islam et al. 2023 and hallucinations in QA Guan et al. 2024 ; Xu et al. 2025 , studies on hallucinations in embodied agents, especially for long-horizon tasks, are limited. Existing efforts Zhou et al. 2024 mainly study incidental cases from generic prompts, capturing only surface-level issues and overlooking the failure patterns behind hallucinations. To fill this gap, we present, to the best of our knowledge, the first empirical study that systematically exemplifies and quantifies hallucinations in long-horizon planning through controlled scene-task inconsistencies. Because existing datasets either do not explicitly aim to elicit hallucination in long-horizon embodied tasks, or do not target embodied agent tasks, we first construct a new probing set that is more likely to cause hallucination. Constructing such a probing set faces two main challenges: (i) how to induce hallucinations and (ii) how to detect unwanted behaviors caused by hallucinations. We solve these challenges by systematically modifying base prompts from the existing LLM-based embodied agent benchmark Li et al. 2024b to introduce scene-task inconsistencies. Specifically, we modify the two core components of each base prompt—the task description and the scene information—to create mismatches between user instructions and the observed environment (see Figure 2 ). With the new hallucination inducing probing set, we conduct an empirical study of popular LLMs. Overall, among our four controlled modifications, Synonymous Object Substitution yields the lowest hallucination rates, suggesting that models can recognize objects conceptually belonging to the same category but often fail to maintain naming consistency, probably due to the inherent tendency of language models to favor varied phrasing over strict lexical alignment. In contrast, the highest hallucination rates occur under Scene Task Contradiction (see Figure 1 ), revealing the model’s inability to ground planning to perceived environments. We also empirically study the effectiveness of mitigation strategies Peng et al. 2023 ; Yin et al. 2024 . The result shows that, even with feedback-based self-correction, hallucinations in Scene Task Contradiction remain high, underscoring a fundamental inability of models to recognize the infeasibility of the task. Additionally, our small-scale experiments with vision-language models (VLMs) suggest that hallucinations are reduced when both image and text inputs are available, emphasizing the importance of cross-modal verification.

Read the complete text