3D-MOOD: Lifting 2D to 3D for Monocular Open-Set Object Detection

Yung-Hsu Yang, Luigi Piccinelli, Mattia Segu, Siyuan Li, Rui Huang, Yuqian Fu, Marc Pollefeys, Hermann Blum, Zuria Bauer · 2025

Read 3D-MOOD: Lifting 2D to 3D for Monocular Open-Set Object Detection by Yung-Hsu Yang, Luigi Piccinelli, Mattia Segu, Siyuan Li, Rui Huang, Yuqian Fu, Marc Pollefeys, Hermann Blum, Zuria Bauer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23567.txt.

Introduction

Yung-Hsu Yang and eight co-authors bring 3D object detection to unfamiliar scenes in a 2025 paper. A single camera is cheap, but teaching a system to locate objects in 3D from one image usually assumes the test world matches training. New rooms and new object types break that assumption.

Their system starts from a 2D detector that already handles open-ended categories, then lifts its finds into 3D with a dedicated box-estimation head. Both stages train together. Geometry cues condition the lifting, and a canonical image space makes training across data sets cleaner. On standard indoor and driving benchmarks, including cross-dataset tests where scenes change entirely, the system set new records. Robots and augmented reality headsets both need this skill: seeing an object once, in a new place, and knowing where it sits in space. Code and models are public.

Catalog record

CREATORYung-Hsu Yang, Luigi Piccinelli, Mattia Segu, Siyuan Li, Rui Huang, Yuqian Fu, Marc Pollefeys, Hermann Blum, Zuria Bauer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-23567

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

3D-MOOD: Lifting 2D to 3D for Monocular Open-Set Object Detection Yung-Hsu Yang, Luigi Piccinelli, Mattia Segu, Siyuan Li, Rui Huang, Yuqian Fu, Marc Pollefeys, Hermann Blum, Zuria Bauer Abstract Monocular 3D object detection is valuable for various applications such as robotics and AR/VR. Existing methods are confined to closed-set settings, where the training and testing sets consist of the same scenes and/or object categories. However, real-world applications often introduce new environments and novel object categories, posing a challenge to these methods. In this paper, we address monocular 3D object detection in an open-set setting and introduce the first end-to-end 3D M onocular O pen-set O bject D etector ( 3D-MOOD ). We propose to lift the open-set 2D detection into 3D space through our designed 3D bounding box head, enabling end-to-end joint training for both 2D and 3D tasks to yield better overall performance. We condition the object queries with geometry prior and overcome the generalization for 3D estimation across diverse scenes. To further improve performance, we design the canonical image space for more efficient cross-dataset training. We evaluate 3D-MOOD on both closed-set settings (Omni3D) and open-set settings (Omni3D → \rightarrow Argoverse 2, ScanNet), and achieve new state-of-the-art results. Code and models are available at royyang0714.github.io/3D-MOOD . 1 Introduction Monocular 3D object detection (3DOD) aims to recognize and localize objects in 3D space from a single 2D image by estimating their 3D positions, dimensions, and orientations. Unlike stereo or LiDAR-based methods, monocular 3DOD relies solely on visual cues, making it significantly more challenging yet cost-effective for robotics and AR/VR applications [ 65 , 10 , 53 , 36 , 16 ] . While many methods [ 51 , 22 , 55 , 28 , 44 , 49 , 63 ] focus on improving 3DOD performance in specific domains, Cube R-CNN [ 4 ] and Uni-MODE [ 23 ] build unified models on the cross-dataset benchmark Omni3D [ 4 ] , which consolidates six diverse 3D detection datasets [ 14 , 5 , 47 , 2 , 1 , 43 ] . These advancements have driven the evolution of 3DOD from specialized models to more unified frameworks. However, as shown in Fig. 1 , most existing methods, including the unified models, operate under an ideal assumption: the training set and testing set share identical scenes and object categories. This limits their generalizability in real-world applications for not being able to detect novel objects in unseen domains. This challenge motivates us to explore monocular open-set 3D object detection, further pushing the boundaries of existing 3DOD methods. Figure 1 : Open-set Monocular 3D Object Detection. Unlike previous methods focusing on achieving good results in the closed-set setting, we aim to resolve the open-set monocular 3D object detection problem. This challenge requires the model to classify arbitrary objects while precisely localizing them in unseen scenes. The first step towards the open-set monocular 3DOD is identifying the fundamental obstacles underlying this task. Our key observations are as follows: 1) Cross-modality learning is crucial to breaking the limitation of closed vocabulary for novel class classification [ 41 ] . However, 3D data lacks rich visual-language pairs, making it significantly more challenging to learn modality alignment and achieve satisfactory open-set results. 2) Robust depth estimation is essential for monocular 3DOD to generalize well across different scenes compared to LiDAR-based methods [ 57 ] . However, monocular depth estimation particularly in novel scenes, is inherently challenging for existing methods. Given the scarcity of 3D data and text pairs, we propose to bridge the modality gap by lifting open-set 2D detection into open-set 3D detection. Fortunately, recent universal metric monocular depth estimation methods [ 38 , 40 , 39 , 3 , 58 ] have shown promising generalization across diverse scenes, which opens new opportunities for addressing open-set monocular 3DOD. Specifically, we design a 3D bounding box head to predict the differentiable lifting parameters from 2D object queries and enable the lift of the detected 2D bounding boxes as 3D object detection. This allows us to jointly train the open-set 2D and 3D detectors in an end-to-end (e2e) way, using both 2D and 3D ground truth (GT). Furthermore, we propose the geometry-aware 3D query generation module, which conditions 2D object queries with the camera intrinsics and depth estimation and generates 3D object queries. These 3D queries encode essential geometric information and are used for the 3D bounding box head to improve the model’s accuracy and generalization ability in 3D object detection. Additionally, we design a more effective canonical image space , which proves crucial for handling datasets with varying image resolutions, as demonstrated in our experiments. Formally, we introduce the first e2e 3D M onocular O pen-set O bject D etecter ( 3D-MOOD ) by integrating the proposed 3D bounding box head, geometry-aware 3D query generation module, and canonical image space into the open-set 2D detector [ 27 ] . Our method takes a monocular input image with the language prompts and outputs the 3D object detection for the desired objects in any given scene. Experimental results demonstrate that 3D-MOOD achieves state-of-the-art (SOTA) performance on the challenging closed-set Omni3D benchmark, surpassing all previous task-specific and unified models. More importantly, in open-set settings, i.e . transferring from Omni3D to Argoverse 2 [ 54 ] and ScanNet [ 8 ] , our method consistently outperforms prior models, achieving clear improvements in generalization and novel classes recognition. Our main contributions are: (1) We explore monocular 3D object detection in open-set settings, establishing benchmarks that account for both novel scenes and unseen object categories; (2) We introduce 3D-MOOD, the first end-to-end open-set monocular 3D object detector, via 2D to 3D lifting, geometry-aware 3D query generation, and canonical image space; (3) We achieve state-of-the-art performance in both closed-set and open-set settings, demonstrating the effectiveness of our method and the feasibility of open-set monocular 3D object detection. Figure 2 : 3D-MOOD. We propose an end-to-end 3D monocular open-set object detector that takes a monocular image and the language prompts of the interested objects as input and classifies and localizes the 3D objects in the scenes. Our design will transform the input image and camera intrinsics into the proposed canonical image space and achieve the open-set ability for diverse scenes. 2 Related Work 2.1 Open-set 2D Object Detection In recent years, there has been tremendous progress in 2D object detection [ 61 , 60 , 27 , 64 , 7 , 13 , 35 ] by leveraging language models [ 9 ] or visual-language foundation models [ 41 ] to detect and classify objects from language queries. Among varying definitions of these works, i.e . open-set object detection , open-world object detection , and open-vocabulary detection , we do not distinguish them in this section and describe the formal problem definition in Sec. 3.1 . OVR-CNN [ 61 ] aligns ResNet [ 15 ] and BERT [ 9 ] features to detect novel classes, while OV-DETR [ 60 ] uses CLIP [ 41 ] image and text embeddings with Detection Transformer [ 6 ] . GLIP [ 27 ] presents grounded language-image pre-training to align object detection and captions. Detic [ 66 ] leverages image level labels to align object reasoning and text to enable tens

Read the complete text