SHREC 2025: Retrieval of Optimal Objects for Multi-modal Enhanced Language and Spatial Assistance (ROOMELSA) Journal: Computers & Graphics

Trong-Thuan Nguyen, Viet-Tham Huynh, Quang-Thuc Nguyen, Hoang-Phuc Nguyen, Long Le Bao, Thai Hoang Minh, Minh Nguyen Anh, Thang Nguyen Tien, Phat Nguyen Thuan, Huy Nguyen Phong, Bao Huynh Thai, Vinh-Tiep Nguyen, Duc-Vu Nguyen, Phu-Hoa Pham, Minh-Huy Le-Hoang, Nguyen-Khang Le, Minh-Chinh Nguyen, Minh · 2025

Read SHREC 2025: Retrieval of Optimal Objects for Multi-modal Enhanced Language and Spatial Assistance (ROOMELSA) Journal: Computers & Graphics by Trong-Thuan Nguyen, Viet-Tham Huynh, Quang-Thuc Nguyen, Hoang-Phuc Nguyen, Long Le Bao, Thai Hoang Minh, Minh Nguyen Anh, Thang Nguyen Tien, Phat Nguyen Thuan, Huy Nguyen Phong, Bao Huynh Thai, Vinh-Tiep Nguyen, Duc-Vu Nguyen, Phu-Hoa Pham, Minh-Huy Le-Hoang, Nguyen-Khang Le, Minh-Chinh Nguyen, Minh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08781.txt.

Introduction

Nguyen and a large team of co-authors launch a benchmark for finding a described object inside a cluttered room and pulling up its 3D model. Most retrieval tests use cropped images and tidy scenes. Real rooms are messy, and people describe targets in loose, free-form words.

Their benchmark, ROOMELSA, spans more than 1,600 apartment scenes and 44,000 queries aimed at spots inside panoramic room views. Rough object matching is largely solved, tests showed. Only one top model put the right match first nearly every time. A light CLIP-based model also did well but tripped on materials and small parts. The authors stress tight vision-language integration.

Catalog record

CREATORTrong-Thuan Nguyen, Viet-Tham Huynh, Quang-Thuc Nguyen, Hoang-Phuc Nguyen, Long Le Bao, Thai Hoang Minh, Minh Nguyen Anh, Thang Nguyen Tien, Phat Nguyen Thuan, Huy Nguyen Phong, Bao Huynh Thai, Vinh-Tiep Nguyen, Duc-Vu Nguyen, Phu-Hoa Pham, Minh-Huy Le-Hoang, Nguyen-Khang Le, Minh-Chinh Nguyen, Minh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-08781

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SHREC 2025: Retrieval of Optimal Objects for Multi-modal Enhanced Language and Spatial Assistance (ROOMELSA) Journal: Computers & Graphics Trong-Thuan Nguyen, Viet-Tham Huynh, Quang-Thuc Nguyen, Hoang-Phuc Nguyen, Long Le Bao, Thai Hoang Minh, Minh Nguyen Anh, Thang Nguyen Tien, Phat Nguyen Thuan, Huy Nguyen Phong, Bao Huynh Thai, Vinh-Tiep Nguyen, Duc-Vu Nguyen, Phu-Hoa Pham, Minh-Huy Le-Hoang, Nguyen-Khang Le, Minh-Chinh Nguyen, Minh-Quan Ho, Ngoc-Long Tran, Hien-Long Le-Hoang, Man-Khoi Tran, Anh-Duong Tran, Kim Nguyen, Quan Nguyen Hung, Dat Phan Thanh, Hoang Tran Van, Tien Huynh Viet, Nhan Nguyen Viet Thien, Dinh-Khoi Vo, Van-Loc Nguyen, Trung-Nghia Le, Tam V. Nguyen, Minh-Triet Tran Abstract Recent 3D retrieval systems are typically designed for simple, controlled scenarios, such as identifying an object from a cropped image or a brief description. However, real-world scenarios are more complex, often requiring the recognition of an object in a cluttered scene based on a vague, free-form description. To this end, we present ROOMELSA, a new benchmark designed to evaluate a system’s ability to interpret natural language. Specifically, ROOMELSA attends to a specific region within a panoramic room image and accurately retrieves the corresponding 3D model from a large database. In addition, ROOMELSA includes over 1,600 apartment scenes, nearly 5,200 rooms, and more than 44,000 targeted queries. Empirically, while coarse object retrieval is largely solved, only one top-performing model consistently ranked the correct match first across nearly all test cases. Notably, a lightweight CLIP-based model also performed well, although it struggled with subtle variations in materials, part structures, and contextual cues, resulting in occasional errors. These findings highlight the importance of tightly integrating visual and language understanding. By bridging the gap between scene-level grounding and fine-grained 3D retrieval, ROOMELSA establishes a new benchmark for advancing robust, real-world 3D recognition systems. 1 Introduction Fig. 1 : Our ROOMELSA challenge. From left to right: a cluttered 3-D apartment rendered as an equirectangular panorama; a binary mask in cyan that highlights the query region; and the gallery-retrieval stage, where the system must rank the single CAD mesh that matches the masked object above all other candidates. The recent proliferation of large-scale indoor scans and publicly available 3D repositories has significantly advanced scene-level understanding, moving beyond the analysis of isolated objects. Tasks such as 3D semantic segmentation, object detection, and visual grounding have benefited substantially from richly annotated datasets, including ScanNet [ 1 ] , Matterport3D [ 2 ] , and 3D-Front [ 3 ] . However, the ability to retrieve a specific 3D object from a cluttered scene based on a natural language description and a spatial mask remains largely underexplored. This functionality is essential for real-world applications ranging from context-aware asset replacement in game engines and augmented reality interior design to robotic manipulation, where a gripper must accurately identify and retrieve a target object amidst similar distractors on a crowded surface. Previous benchmarks addressing language–vision alignment in 3D environments, such as ReferIt3D [ 4 ] , ScanRefer [ 5 ] , and ViGiL3D [ 6 ] , typically formulate the task as object localization. Given a natural language description, the objective is to predict a bounding box or point-cloud cluster corresponding to the referred object within a 3D scan. In contrast, ROOMELSA (Retrieval of Optimal Objects for Multi-modal Enhanced Language and Spatial Assistance) introduces a reversed formulation. Instead of inferring spatial location from language, ROOMELSA assumes a predefined spatial region, specified by a mask, and requires retrieving the 3D model from a large-scale object catalog that best matches the semantics and geometry of the masked region. This reformulation introduces three principal challenges, as illustrated in Fig. 1 . First, the task presents a significant cross-domain appearance gap, as RGB-D renderings of real-world, cluttered scenes differ markedly from the clean, studio-lit images used to depict catalog exemplars. Second, it demands fine-grained visual discrimination. Especially, many scenes contain multiple instances from the same object category, such as dining chairs, and accurate retrieval hinges on the system’s ability to discern subtle variations in shape, material, and style that are often only implicitly described in language. Third, effective retrieval requires robust spatial reasoning, as referring expressions often contain relational cues (e.g., “the mug closest to the sink” or “the lamp behind the sofa”), which demand a holistic understanding of both the masked region and the broader spatial configuration of the RGB-D rendered scene. To foster progress in this emerging direction, we introduce the ROOMELSA Challenge as part of SHREC 2025 1 1 1 https://www.shrec.net/ and the first benchmark dataset explicitly designed for this task. The dataset comprises 1,622 apartment-level scenes, encompassing 5,197 individual rooms, and includes a total of 44,445 query pairs (mask, text). The data is divided into public and private test splits. For each masked query, participants were asked to submit a ranked list of ten candidate catalog item IDs, with performance evaluated using the Mean Reciprocal Rank (MRR). Unlike previous challenges centered on generic retrieval or phrase grounding [ 7 , 8 , 9 , 10 , 11 , 12 , 13 , 14 ] , our ROOMELSA challenge introduces three distinctive features. It conditions language queries on predefined spatial masks, includes a significantly broader taxonomy of furniture and décor styles, and supports a zero-shot public test phase to encourage scalable, generalizable solutions without extensive fine-tuning. The 2025 edition attracted 18 teams from across the globe, who explored various algorithmic strategies, including frozen vision–language models, depth-aware point-cloud alignment, multi-view CLIP indexing, mask-guided inpainting, and hybrid voting mechanisms. The top-performing method achieved an MRR of 0.97, setting a benchmark for ROOMELSA and future work. Contributions. This paper makes three main contributions: 1. In the ROOMELSA challenge, we formulate 3D object retrieval as a mask-conditioned, language-driven grounding task in 3D environments, unifying spatial reference and natural language understanding within a unified task. 2. We introduce ROOMELSA 2 2 2 The ROOMELSA dataset is released at https://aichallenge.hcmus.edu.vn/shrec-2025/smart3droom , a novel benchmark dataset consisting of 1,622 apartment-scale scenes and 5,197 individual rooms, with 44,445 (mask, text) query pairs. 3. We analyze the top-five challenge entries, revealing how scene-level inference, depth-aware reconstruction, and adaptive cross-modal fusion affect Mean Reciprocal Rank. The remainder of this paper is organized as follows. Sec. 2 reviews related work on language-grounded object retrieval in 3D, referring segmentation, and multimodal foundation models for 3D understanding. Sec. 3 discusses the limitations of previous challenges and highlights the advantages of the proposed ROOMELSA challenge. Sec. 4 introduces the ROOMELSA dataset, defines the task, and describes the evaluation metrics. Sec. 5 provides an overview of participant statistics and summarizes the submitted approaches. Sec. 6 presents the methods proposed by the top five participating teams. Sec. 7 reports both quantitative and qualitative results, followed by a detailed analysis. Finally, Sec. 8 concludes the paper and outlines future research directions enabled by the ROOMELSA benchmark. Fig. 2 : Samples from our dataset for the ROOMELSA challenge, part of SHREC 2025. Each example includes the rendered scene with the object masked in cyan , the corresponding text query provided to the retrieval model, and the ground-truth 3D mesh (e.g., refrigerator, nightstand) rendered from a canonical viewpoint. 2 Related Work 2.1 Language-grounded Object Retrieval in 3D Language-grounded object retrieval in 3D has evolved from single-object detection to more complex set-level grounding. Specifically, ScanRefer

Read the complete text