: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarios

Yifan Li, Yuhang Chen, Anh Dao, Lichi Li, Zhongyi Cai, Zhen Tan, Tianlong Chen, Yu Kong · 2025

Read : Pushing the Frontiers of Embodied Question Answering in Industrial Scenarios by Yifan Li, Yuhang Chen, Anh Dao, Lichi Li, Zhongyi Cai, Zhen Tan, Tianlong Chen, Yu Kong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20640.txt.

Introduction

Yifan Li and co-authors wrote a paper from 2025 that sets a new test for robots that answer questions while moving through a place. Past tests of this kind used homes. Real work sites, such as warehouses, add safety risks that homes lack. The team built their test inside a detailed simulator of warehouse scenes.

The scenes hold varied gear, moving workers, and staged hazards drawn from real safety rules. Questions come in six kinds, from naming gear to judging time and space. The set holds close to one thousand pairs from small warehouses and several hundred from large ones, with and without people present. The team also adds a check on the steps of thought behind each answer. They ran a range of base models through the test to gauge how ready such agents are for real sites.

Catalog record

CREATORYifan Li, Yuhang Chen, Anh Dao, Lichi Li, Zhongyi Cai, Zhen Tan, Tianlong Chen, Yu Kong
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20640

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

: Pushing the Frontiers of Embodied Question Answering in Industrial Scenarios Yifan Li, Yuhang Chen, Anh Dao, Lichi Li, Zhongyi Cai, Zhen Tan, Tianlong Chen, Yu Kong Abstract Existing Embodied Question Answering (EQA) benchmarks primarily focus on household environments, often overlooking safety-critical aspects and reasoning processes pertinent to industrial settings. This drawback limits the evaluation of agent readiness for real-world industrial applications. To bridge this, we introduce IndustryEQA, the first benchmark dedicated to evaluating embodied agent capabilities within safety-critical warehouse scenarios. Built upon the NVIDIA Isaac Sim platform, IndustryEQA provides high-fidelity episodic memory videos featuring diverse industrial assets, dynamic human agents, and carefully designed hazardous situations inspired by real-world safety guidelines. The benchmark includes rich annotations covering six categories: equipment safety, human safety, object recognition, attribute recognition, temporal understanding, and spatial understanding. Besides, it also provides extra reasoning evaluation based on these categories. Specifically, it comprises 971 question-answer pairs generated from small warehouse and 373 pairs from large ones, incorporating scenarios with and without human. We further propose a comprehensive evaluation framework, including various baseline models, to assess their general perception and reasoning abilities in industrial environments. IndustryEQA aims to steer EQA research towards developing more robust, safety-aware, and practically applicable embodied agents for complex industrial environments. Benchmark 1 1 1 https://huggingface.co/datasets/IndustryEQA/IndustryEQA and codes 2 2 2 https://github.com/JackYFL/IndustryEQA are available. 1 Introduction Embodied Artificial Intelligence (AI) aims to develop agents that perceive, reason, and interact intelligently within the physical world. A key aspect is understanding from an egocentric view by interpreting open-ended language grounded in common-sense knowledge. Evaluating such complex environmental understanding continues to pose a substantial challenge. Embodied Question Answering (EQA) [ 7 ] has emerged as a comprehensive task for this evaluation, requiring an agent to explore its surrounding environment, collect visual evidence, and answer natural language questions. This task encompasses multiple essential capabilities, including episodic memory, purposeful navigation, visual perception, spatial reasoning, etc . Recently, studies such as OpenEQA [ 23 ] have begun investigating the open-vocabulary EQA problem within more realistic settings. Figure 1 : An illustration of the IndustryEQA benchmark, consisting of episodic memory videos and annotations. IndustryEQA annotations incorporate six types of annotations, covering safety (equipment safety and human safety) and general perception capabilities (object recognition, attribute recognition, temporal understanding and spatial understanding). Furthermore, it also incorporates extra reasoning answers for the questions that require deeper thinking. Table 1 : Comparison with current EQA benchmarks. “Human” indicates whether the benchmark includes human subjects, and “EM” stands for episodic memory. Benchmark Scenario Platform Safety Reasoning Open Vocab Human LLM scoring EM Video EQA-v1 CVPR18 [ 7 ] Household House3D ✗ ✗ ✗ ✗ ✗ ✗ IQUAD CVPR18 [ 13 ] Household AI2-THOR ✗ ✗ ✗ ✗ ✗ ✗ MP3D-EQA CVPR19 [ 44 ] Household Matterport3D ✗ ✗ ✗ ✗ ✗ ✗ MT-EQA CVPR19 [ 49 ] Household House3D ✗ ✗ ✗ ✗ ✗ ✗ ScanQA CVPR22 [ 2 ] Household RGB-D Camera ✗ ✗ ✗ ✗ ✗ ✓ SQA3D ICLR23 [ 22 ] Household ScanNet ✗ ✗ ✗ ✗ ✗ ✗ K-EQA TPAMI23 [ 39 ] Household AI2-THOR ✗ ✗ ✓ ✗ ✗ ✗ Robo-VQA ICRA24 [ 36 ] Household RGB Camera ✗ ✗ ✗ ✗ ✗ ✓ OpenEQA CVPR24 [ 23 ] Household ScanNet/HM3D ✗ ✗ ✓ ✗ ✓ ✓ CityEQA-EC ArXiv25 [ 50 ] City EmbodiedCity ✗ ✓ ✓ ✗ ✓ ✗ IndustryEQA Warehouse Isaac Sim ✓ ✓ ✓ ✓ ✓ ✓ However, as shown in Tab. 1 , current EQA research predominantly considers the household scenarios (like kitchens or rooms) by utilizing simulated household environments ( e.g. , House3D [ 45 ] , AI2-THOR [ 15 ] , Habitat [ 25 , 38 , 32 ] ). These tasks usually involve general understanding tasks like identifying objects, attributes, or spatial relationships, etc. , requiring short phrasal responses. While these benchmarks are instrumental in advancing foundational capabilities, they still exhibit certain limitations. Firstly, their focus on specific indoor scenarios limits applicability to distinct environments, such as industrial scenarios. Secondly, they primarily emphasize general capabilities, while overlooking specific perspectives like safety within the environment. Thirdly, most of them lack the procedural or causal process after giving the short phrasal answer, which leads to insufficient evaluation of the agent’s reasoning ability. These constraints limit the effectiveness of EQA in evaluating agent readiness for real-world applications. From practical angle, directly collecting industrial data from real warehouses raises concerns related to costs, privacy, safety, and liability. It also offers limited controllability and diversity over the various assets within the environment. To address these shortcomings and steer EQA research towards industrial scenarios, we introduce IndustryEQA (see Fig. 1 ), the first benchmark dedicated to evaluating embodied agent capabilities in the safety-critical context of industrial warehouses. IndustryEQA moves beyond the limitations of existing EQA benchmarks by providing a challenging evaluation focused on industrially relevant perception, multifaceted reasoning, and crucial safety awareness. Our benchmark is built on the NVIDIA Isaac Sim platform [ 24 ] , which provides high-fidelity physics simulation and supports flexible, diverse warehouse designs. This powerful simulation engine enables the creation of realistic industrial scenarios that include human agents—an element largely overlooked in previous EQA benchmarks. Furthermore, as shown in Fig. 1 , we curate question–answer annotations across six categories to assess two crucial abilities: safety awareness and general perception. In addition to evaluating the models’ direct answers, we also include reasoning annotations to examine their reasoning capabilities. The overall contributions are summarized as follows: • We take a significant step forward by extending the household EQA task to an industry-oriented setting, enabling the embodied AI community to evaluate agents’ safety-aware understanding and reasoning capabilities; • We propose the first industrial EQA benchmark IndustryEQA , which consists of episodic memory videos collected from the Isaac Sim simulator, along with question–answer annotations spanning six categories, accompanied by reasoning-based answers; • We design a comprehensive evaluation framework and assess multiple baseline models on IndustryEQA. A detailed analysis of the results is provided, offering key insights into model performance. 2 Related Work Embodied Question Answering (EQA). The EQA task was first introduced by Das et al. [ 7 ] , which is defined as requiring an agent to actively explore the environment and perform atomic actions based on egocentric visual perception. Later studies, say ScanQA [ 2 ] , RoboVQA [ 36 ] , and OpenEQA [ 23 ] , extend this active setting to episodic memory one, which collects all the visual information and then perform question answering based on the visual perception. Such a setting is similar to visual question answering (VQA), where a model is required to answer natural language questions given visual inputs such as images or videos. A key distinction is that EQA tasks primarily focus on the egocentric view and are not limited to a single question type. Our IndustryEQA takes the episodic memory setting, which is easy to benchmark current visual large language models (VLLMs) [ 19 ] . Due to concerns regarding cost, privacy, controllability, and diversity, most existing EQA benchmarks rely on simulators to generate environments. However, almost all of these environments focus on home scenarios like kitchens or rooms, leaving domains like industrial settings largely underexplored. Our proposed IndustryEQA aims

Read the complete text