Open-Vocabulary 3D Instruction Ambiguity Detection
Jiayu Ding, Haoran Tang, Ge Li · 2026
Read Open-Vocabulary 3D Instruction Ambiguity Detection by Jiayu Ding, Haoran Tang, Ge Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-05991.txt.
Introduction
In a 2026 paper, Jiayu Ding and 4 co-authors define a new safety task for robots and other embodied systems. A command given in a three-dimensional scene may point to more than one object. The model must judge whether the instruction has only one clear meaning before anyone acts on it.
To test this ability the team built Ambi3D, a benchmark with over 700 scenes and about 22,000 instructions. Leading 3D language models struggled to tell clear commands from vague ones. Their AmbiVer framework gathers visual evidence from several views and guides a vision-language model in its judgment. Tests show AmbiVer works better, pointing toward safer embodied systems that pause and check.
Catalog record
| CREATOR | Jiayu Ding, Haoran Tang, Ge Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-05991
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Open-Vocabulary 3D Instruction Ambiguity Detection Jiayu Ding, Haoran Tang, Ge Li Abstract In safety-critical domains, linguistic ambiguity can have severe consequences; a vague command like “Pass me the vial” in a surgical setting could lead to catastrophic errors. Yet, most embodied AI research overlooks this, assuming instructions are clear and focusing on execution rather than confirmation. To address this critical safety gap, we are the first to define Open-Vocabulary 3D Instruction Ambiguity Detection, a fundamental new task where a model must determine if a command has a single, unambiguous meaning within a given 3D scene. To support this research, we build Ambi3D, the large-scale benchmark for this task, featuring over 700 diverse 3D scenes and around 22k instructions. Our analysis reveals a surprising limitation: state-of-the-art 3D Large Language Models (LLMs) struggle to reliably determine if an instruction is ambiguous. To address this challenge, we propose AmbiVer, a two-stage framework that collects explicit visual evidence from multiple views and uses it to guide an vision-language model (VLM) in judging instruction ambiguity. Extensive experiments demonstrate the challenge of our task and the effectiveness of AmbiVer, paving the way for safer and more trustworthy embodied AI. Code and dataset available at https://jiayuding031020.github.io/ambi3d/ . 1 Introduction The reliability of an agent’s interaction with the physical world heavily depends on the precision of its understanding of human instructions. Consider a safety-critical robotics scenario (see Figure 1 ) : a surgeon instructs a robot, “Pass me the vial from the tray” , when a bottle of benign herbal extract and a lethal anesthetic are placed side-by-side. A system incapable of recognizing and resolving this instructional ambiguity might select an object arbitrarily, with potentially catastrophic consequences. Such safety risks, arising from linguistic ambiguity, are prevalent in numerous domains, including home services, industrial automation, and Augmented Reality (AR) assisted operations. A trustworthy intelligent system must first confirm its interpretation is unambiguous before proceeding with execution. Figure 1 : This high-stakes scenario highlights a critical safety challenge where an ambiguous instruction forces a robot to choose between a harmless substance and a lethal one. Despite the critical importance of this challenge, the research focus in embodied intelligence has predominantly centered on “grounding” language in vision and subsequent “execution” . While this paradigm has achieved remarkable success in many tasks, its inherent unambiguous instruction assumption introduces significant latent risks. Visual Question Answering (VQA), for instance, has significantly advanced multimodal understanding, but its models are often optimized on datasets where questions are presumed to have a single, verifiable answer. For example, in a smart home context, confronted with the question, “Was the stove in the kitchen turned off when I left?” a VQA model might observe that the main stovetop is off and answer “Yes” , overlooking a portable induction cooktop still operating in a corner. This affirmative answer creates a false sense of security, rooted in the model’s inability to recognize that the scope of “the stove in the kitchen” is itself ambiguous. Fundamentally, these models are designed to find the “right answer” to an input assumed to be valid, lacking an intrinsic mechanism to identify when the input itself is a “bad question” . This challenge is exacerbated in complex 3D environments, where referents may be partially occluded and their spatial relationships are often view-dependent. A truly reliable system must possess the ability to recognize such ambiguity and proactively seek clarification, rather than blindly guessing. This tendency toward blind guessing reflects a longstanding bias in embodied AI research: an emphasis on the correctness of instruction execution while overlooking the executability of the instruction itself. Recent studies have begun to address linguistic ambiguities arising in human–agent interaction, proposing two broad classes of resolution strategies. The first is passive resolution , which makes a “best guess” under uncertainty: the agent either infers the most plausible target from contextual priors Chen et al. 2020 ; Magassouba et al. 2018 or executes a tentative action and relies on human feedback for correction Majumdar et al. 2023 ; Dai et al. 2024 . The second is active clarification Ren et al. 2023 ; Taioli et al. 2025 , where the agent proactively queries the user when recognizing low confidence or uncertain actions. However, both paradigms base ambiguity judgments on the model’s internal subjective state . As a result, a model may express high confidence in an objectively ambiguous instruction yet hesitate over a clear one. Consequently, the agent lacks the foundational ability to first ask: “Is this instruction objectively ambiguous given this specific 3D environment?” To systematically address this fundamental safety problem, we are the first to formally define the new task of Open-Vocabulary 3D Instruction Ambiguity Detection . The task requires a model to take a 3D scene and an open-vocabulary natural language instruction as input, and to determine whether the instruction is ambiguous. To facilitate systematic research on this new task, we construct Ambi3D , the large-scale benchmark featuring a meticulously human-annotated set of instructions that capture complex referential and execution ambiguities grounded in real-world scenes. However, our analysis reveals a surprising limitation in existing methods: state-of-the-art 3D LLMs struggle to reliably determine if an instruction is ambiguous. To overcome this limitation, we propose Ambi guity Ver ifier (AmbiVer), a two-stage framework that decouples scene perception from logical reasoning. Its perception stage converts the raw scene and instruction into a set of structured evidence. The reasoning stage then passes this evidence to a zero-shot VLM for logical adjudication. In summary, our contributions are as follows: • We define and formalize the critical safety task of Open-Vocabulary 3D Instruction Ambiguity Detection. • We build Ambi3D , a large-scale benchmark for this task, featuring \sim 22k human-annotated instructions grounded in over 700 diverse real-world 3D scenes. • We propose AmbiVer , a novel two-stage framework that decouples perception from logical reasoning, leveraging a VLM for zero-shot, evidence-based adjudication. 2 Related Works Linguistic Ambiguity Linguistic ambiguity is a fundamental challenge in Natural Language Processing (NLP), stemming from words or phrases corresponding to multiple meanings Berry and Kamsties 2004 ; Yadav et al. 2021 . Prior work has primarily focused on ambiguity arising from language-internal factors, branching into several key directions. One direction is lexical ambiguity Pethö 2001 ; Tabanakova et al. 2021 ; Abeysiriwardana and Sumanathilaka 2024 , which refers to a string of sounds or characters corresponding to multiple lexical or semantic interpretations. Another research avenue is syntactic ambiguity Abeysiriwardana and Sumanathilaka 2024 . This ambiguity arises not from individual words but from the grammatical structure of a sentence or phrase, leading to multiple valid parsing interpretations for the entire sentence. Additionally, other types of ambiguity, such as semantic ambiguity Poesio 1995 and phonological ambiguity Frost et al. 1990 , have also been explored. However, this work is different from our research, as it concentrates on language-internal ambiguity rooted in lexicon or syntax. In contrast, we focus on grounded instructional ambiguity, which is a property jointly determined by the instruction and the 3D scene. Ambiguity Resolution in Embodied AI. Although prior work in embodied AI has recognized the issue of linguistic ambiguity, research has largely focused on resolving rather than detecting it. Existing approaches generally fall into two paradigms. The first, passive resolution ,