EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models

Haozhe Shan, Xiancong Ren, Han Dong, Haoyuan Shi, Yingji Zhang, Jiayu Hu, Yi Zhang, Yong Dai, Bin Shen, Lizhen Qu, Zenglin Xu, Xiaozhu Ju · 2026

Read EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models by Haozhe Shan, Xiancong Ren, Han Dong, Haoyuan Shi, Yingji Zhang, Jiayu Hu, Yi Zhang, Yong Dai, Bin Shen, Lizhen Qu, Zenglin Xu, Xiaozhu Ju (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17070.txt.

Introduction

Embodied agents increasingly rely on VLMs as their perceptual backbone, yet how those models ground language in embodied settings remains unexplored. Shan's team fills that gap with EPIC-Bench, a perception-centred benchmark for fine-grained embodied visual grounding in vision-language models. It holds 6.6 thousand human-annotated tuples of image, text, and mask. They span three stages of embodied interaction: target localisation, navigation, and manipulation. With 23 fine-grained tasks and multi-dimensional settings, evaluation over 80 leading models reportedly yields a diagnostic analysis of grounding ability.

Catalog record

CREATORHaozhe Shan, Xiancong Ren, Han Dong, Haoyuan Shi, Yingji Zhang, Jiayu Hu, Yi Zhang, Yong Dai, Bin Shen, Lizhen Qu, Zenglin Xu, Xiaozhu Ju
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55189
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17070

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

EPIC-Bench: A Perception-Centric Benchmark for Fine-Grained Embodied Visual Grounding in Vision-Language Models Haozhe Shan, Xiancong Ren, Han Dong, Haoyuan Shi, Yingji Zhang, Jiayu Hu, Yi Zhang, Yong Dai, Bin Shen, Lizhen Qu, Zenglin Xu, Xiaozhu Ju Abstract While large vision-language models (VLMs) are increasingly adopted as the perceptual backbone for embodied agents, existing benchmarks often rely on question-answering or multiple-choice formats. These protocols allow models to exploit linguistic priors rather than demonstrating genuine visual grounding. To address this, we present EPIC-Bench , E mbodied P ercept I on Ben C hmark, a fine-grained grounding benchmark designed to systematically evaluate the visual perceptual capabilities of VLMs in real-world embodied environments. Comprising 6.6k meticulously annotated tuples (Image, Text, Mask), EPIC-Bench spans 23 fine-grained tasks across three core stages of the embodied interaction pipeline: Target Localization, Navigation, and Manipulation. Extensive evaluations of over 89 leading VLMs reveal that while advanced reasoning models show promise, current VLMs universally struggle with complex visual-text alignment for physical interactions. Specifically, models exhibit critical bottlenecks in multi-target counting, part-whole relationship understanding, and affordance region detection. EPIC-Bench provides a robust foundation and actionable insights for advancing the next generation of vision-driven embodied models. Figure 1: Overview of EPIC-Bench. The benchmark evaluates embodied visual perception through mask-grounded tasks spanning target localization, navigation-oriented perception, and manipulation-oriented perception. Unlike QA or MCQ, EPIC-Bench requires models to localize task-relevant objects, regions, paths, and affordance areas in real-world embodied scenes. It contains 6,661 human-annotated samples across 23 tasks and supports large-scale evaluation of 89 representative VLMs. 1 Introduction Vision is the primary modality through which embodied agents perceive the physical world, forming the foundation for downstream reasoning and planning. With the rapid advancement of large vision–language models (VLMs), these models have increasingly been adopted as the perceptual backbone of embodied systems 72 ; 73 ; 60 ; 56 ; 61 , and corresponding benchmarks have been proposed to evaluate their capabilities in perception, reasoning, and planning 60 ; 20 ; 54 ; 23 ; 68 ; 17 . Nevertheless, a fundamental question remains: Can current VLMs generalize to real-world embodied visual perception tasks? Answering this question requires systematic benchmarking to understand the extent to which existing VLMs can support embodied deployment and to identify the perceptual capabilities that still require improvement. Embodied visual perception requires not only recognizing the existence of a target object but also precisely determining its spatial location to support downstream tasks such as navigation and manipulation. However, traditional visual benchmarks commonly rely on question-answering (QA) 38 ; 68 ; 28 or multiple-choice (MCQ) formats 20 ; 60 ; 27 . Such protocols may yield overly optimistic evaluations of embodied perception, as models can exploit linguistic priors and common-sense reasoning instead of demonstrating genuine visual grounding ability. The detailed comparison is represented in Tab. 1 . Within embodied visual perception, visual grounding, the process of localizing language instructions with specific objects and spatial locations in the physical environment, is a fundamental capability. However, existing visual grounding benchmarks only partially capture this requirement by predominantly focusing on object detection in generic scenes 70 ; 67 ; 50 . Many tasks reduce to category-level retrieval with relatively simple textual descriptions or lack task-oriented perception requirements that are critical in embodied environments 35 ; 70 . Although recent embodied perception benchmarks have begun to examine specific capabilities, such as spatial relation understanding and free-space detection 27 ; 54 ; 20 , a comprehensive and systematic evaluation of embodied visual perception remains lacking. To bridge this gap, we introduce EPIC-Bench , a comprehensive benchmark for embodied visual perception comprising 6,661 testing instances organized into 9 subcategories and 23 tasks. It covers the fine-grained perception pipeline required for embodied agents: from localizing the target object upon receiving an instruction, to reasoning about navigation toward it, and ultimately supporting task-specific manipulation. We adopt mask grounding as the primary evaluation protocol, complemented by three additional scoring metrics to provide multi-dimensional assessment. The key contributions of our paper can be summarized below: • We introduce EPIC-Bench , a large-scale benchmark specifically designed to evaluate mask-level embodied visual perception in VLMs. It comprises 6,661 testing instances across 9 subcategories and 23 tasks. To ensure annotation quality, we employ 20 human annotators with undergraduate-level education, accumulating over 4,800 person-hours of annotation effort across 30 working days. The benchmark is publicly available. • The evaluation framework centered on mask grounding mitigates shortcut exploitation from language priors and better reflects perception requirements in embodied environments. • We conduct extensive experiments and ablation studies on a diverse set of 89 representative VLMs. Our findings provide actionable insights into current limitations and offer guidance for future research on embodied downstream improvement. 2 Related Work Vision Language Models for Embodied Tasks. A growing body of work leverages VLMs to build embodied systems. Gemini Robotics-ER 59 ; 58 extends Gemini’s multimodal reasoning capabilities into the physical world. Many recent studies have worked on language-guide task like navigation and manipulation, LERF 30 queries conditions on object masking to separate sub-parts of the object. ShapeGrasp 33 infers contact points by prompting the VLMs via Chain of Thought. Visual Grounding. Visual grounding requires models to localize target objects based on textual descriptions. RefCOCO-test 70 is among the most representative benchmarks, though its annotations assume exactly one target per description. D^{3} 67 and OmniLabel 50 extend to complex language-based detection but adopt bounding-box annotations, which are less suitable for embodied scenarios requiring fine-grained localization of irregular objects. GRES 35 introduces mask-level annotations; however, it focuses on generic scene understanding rather than embodied perception tasks. Embodied Benchmarks. Several benchmarks have been proposed to evaluate visual perception capabilities in embodied scenarios. EmbSpatial 20 constructs template-based MCQs to assess model’s understanding of six canonical spatial relations. RoboSpatial-Home 54 and RefSpatial-Bench 75 adopts point selection and binary-choice formats to evaluate spatial relation reasoning and object placement understanding. OpenEQA 38 and ERQA 60 evaluate multimodal understanding abilities associated with target localization and manipulation-oriented reasoning. RoboAfford-Eval 23 focuses on localization capabilities relevant to grasping operations, along with global target localization and free-space identification. Overall, these benchmarks predominantly adopt QA, MCQ, or point-based evaluation formats. Moreover, they typically assess only a subset of the capabilities required to complete a full embodied instruction pipeline. Consequently, they fall short of providing a comprehensive evaluation of embodied visual perception: an important gap that our work aims to address. A detailed comparison of existing benchmarks is provided in Tab. 1 . Benchmark / Dataset GT Type Target Localization Navigation Manipulation Data Domain Manual Multi View Size BA SRA ECA GD FP VM AR CR PR EmbSpatial-Bench 20 MCQ ✓ ✓ Ind. 3.6k RoboSpatial-Home 54 Point+Binary ✓ ✓ ✓ Ind. ✓ 350 RefSpatial-Bench 75 Point/Mask ✓ ✓ ✓ Gen./Ind. ✓ 200 OmniSpatial 27 MCQ ✓ ✓ ✓ † ✓ Gen./Ind. ✓ 8.4k OpenEQA 38 QA ✓ ✓ ✓ ✓ Ind./Ego. ✓ ✓ 1.6k ERQA 60 MCQ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Ind./Ego./Robo ✓ ✓ 400 RoboAfford-Eval 23 Point ✓ ✓ † ✓ † ✓ ✓ Gen./Ind./Robo./Ego. ✓ 338 EmbodiedBench 68 QA ✓ ✓ † ✓ ‡ ✓ ✓ ✓ Ind./Robo. ✓ ‡ ✓ 1.1k EXPRESS-Bench 28 QA ✓ ✓ ✓ Ind. ✓ 2.0k RynnBrainBench 17 QA+Point ✓ ✓ ✓ ✓ ✓

Read the complete text