Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3D

Daniel DeTone, Tianwei Shen, Fan Zhang, Lingni Ma, Julian Straub, Richard Newcombe, Jakob Engel · 2026

Read Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3D by Daniel DeTone, Tianwei Shen, Fan Zhang, Lingni Ma, Julian Straub, Richard Newcombe, Jakob Engel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05212.txt.

Introduction

DeTone and six colleagues present Boxer, an algorithm that lifts 2D bounding boxes to 3D boxes from posed images and depth. At its core, BoxerNet, a network, lifts 2D proposals into 3D, then fuses views and filters geometry to give consistent boxes in world space. By reusing 2D detectors, BoxerNet focuses on lifting rather than detecting, cutting demand for annotated 3D training data. Trained on over 1.2 million 3D boxes, it beats CuTR in settings without depth, 53 to 1 on mAP, and on CA-1M with depth, 41 to 25 on mAP.

Catalog record

CREATORDaniel DeTone, Tianwei Shen, Fan Zhang, Lingni Ma, Julian Straub, Richard Newcombe, Jakob Engel
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE49810
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05212

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Boxer: Robust Lifting of Open-World 2D Bounding Boxes to 3D Daniel DeTone, Tianwei Shen, Fan Zhang, Lingni Ma, Julian Straub, Richard Newcombe, Jakob Engel Abstract Detecting and localizing objects in space is a fundamental computer vision problem. While much progress has been made to solve 2D object detection, 3D object localization is much less explored and far from solved, especially for open-world categories. To address this research challenge, we propose Boxer, an algorithm to estimate static 3D bounding boxes (3DBBs) from 2D open-vocabulary object detections, posed images and optional depth either represented as a sparse point cloud or dense depth. At its core is BoxerNet, a transformer-based network which lifts 2D bounding box (2DBB) proposals into 3D, followed by multi-view fusion and geometric filtering to produce globally consistent de-duplicated 3DBBs in metric world space. Boxer leverages the power of existing 2DBB detection algorithms ( e.g., DETIC [ 54 ] , OWLv2 [ 31 ] , SAM3 [ 8 ] ) to localize objects in 2D. This allows the main BoxerNet model to focus on lifting to 3D rather than detecting, ultimately reducing the demand for costly annotated 3DBB training data. Extending the CuTR [ 21 ] formulation, we incorporate an aleatoric uncertainty for robust regression, a median depth patch encoding to support sparse depth inputs, and large-scale training with over 1.2 million unique 3DBBs. BoxerNet outperforms state-of-the-art baselines in open-world 3DBB lifting, including CuTR in egocentric settings without dense depth (0.532 vs. 0.010 mAP) and on CA-1M with dense depth available (0.412 vs. 0.250 mAP). Project page with code available here: https://facebookresearch.github.io/boxer . Figure 1 : Boxer takes as input posed images with optional depth and off-the-shelf 2DBB open-world detections, estimating static, global 3D bounding boxes. Boxer is run on a sequence and various scenes are highlighted to show the accuracy and open-world coverage of objects such as spice jar, hairdryer, sink drain and TV remote. 1 Introduction The transition from passive observation to active interaction in contextual AI and embodiment requires engaging the metric 3D world. Estimating 3D bounding boxes (3DBBs) for arbitrary objects is central to this process, providing the semantic and geometric foundation for spatial understanding. An agent must reason about object identity, scale, orientation, and proximity to support goal-directed autonomous navigation and dexterous manipulation. The same capability is critical for immersive augmented reality (AR) and digital twins, where AI must align semantic knowledge with physical space for seamless human-machine collaboration. Despite being a cornerstone of contextual AI and embodiment, 3DBB estimation remains fraught with unresolved challenges. The primary obstacle is a profound data-annotation disparity. While 2D detection scales through intuitive image-space labeling [ 24 , 20 ] , metric 3DBB estimation from a monocular view is inherently ill-posed. Collecting and annotating 3DBBs therefore requires specialized setups, typically involving LiDAR, depth sensors, or calibrated multi-view systems for 3D grounding. This creates a bottleneck, leaving even the largest 3D datasets orders of magnitude smaller than their 2D counterparts [ 6 , 21 ] . As a result, 3D models are often starved of the “web-scale” diversity that has enabled 2D vision-language models to learn the long tail of open-world objects. Furthermore, the current 3DBB data landscape is fragmented across sensors and modalities. Available datasets form a patchwork: some provide dense depth maps from RGB-D cameras, while others provide sparse point clouds from simultaneous localization and mapping (SLAM) [ 13 ] or structure-from-motion (SfM) [ 42 ] . Data are also captured with diverse camera models, ranging from pinhole to fisheye and panoramic lenses. The data and annotation landscape leaves the field under-explored and technologically rigid. Some methods remain trapped in a “closed-set” taxonomy, with no clear path to the open-world long tail [ 50 , 52 ] , as they rely heavily on closed-set 3D supervision [ 6 ] . Other approaches rely heavily on 3D supervision for both detection and localization, placing high demands on training data and failing to leverage mature 2D detection as a bootstrap [ 44 , 21 ] . Moreover, current 3D estimation paradigms struggle to handle this heterogeneity because camera models and sensor modalities are often baked into the architecture, requiring substantial re-engineering or retraining to transfer across settings. In summary, there is still no universal interface for 3D perception that bridges powerful 2D semantics and metric 3D geometry. Such a system should adapt its predictions to available geometric cues, whether from dense geometry or sparse points, while remaining agnostic to the underlying camera model. Motivated by this gap, we bridge 2D and 3D detection by decomposing the problem into two stages: 2D localization followed by metric 3D lifting. By using existing open-world 2D detectors trained on internet-scale data, we inherit broad semantic coverage and strong localization in the image plane. We then focus model capacity on the geometric lifting step, learning to predict metric 3D bounding boxes from 2D proposals and depth cues. As illustrated in Fig. 1 , this decomposition allows us to benefit from the scale of 2D supervision while addressing the geometric requirements of 3D detection. Additionally, we use flexible input conditioning to support training on pinhole and fisheye camera models and on datasets with and without sparse point clouds or dense depth maps. The contributions of this work are summarized as follows. • We propose Boxer , a complete algorithm for estimating open-world, global, de-duplicated 3D bounding boxes for objects using posed, calibrated video with optional sparse point clouds or dense depth. • We introduce BoxerNet , a transformer-based model that lifts 2D bounding box detections from off-the-shelf open-world detectors into metric 3D bounding boxes. It improves upon the CuTR [ 21 ] architecture by incorporating an aleatoric uncertainty head and a median depth patch encoding which enables large-scale training on over 1.22 million unique 3DBBs across four different device types. • BoxerNet outperforms state-of-the-art baselines in the task of open-world 2D to 3D bounding box lifting across different camera types, including outperforming CuTR in egocentric settings without dense depth (0.532 vs. 0.010 mAP) and on CA-1M with dense depth available (0.412 vs. 0.250 mAP). • We release code and model to enable inference on a variety of input data. 2 Related Work We review closed-set and open-set 3D object detection methods as well as, briefly, open-set 2D bounding box detection since Boxer builds on these. Closed-world 3D detection. Many approaches for 3DBB detection exist that focus on a fixed set of classes (e.g. chair, table, lamp) such as Cube-RCNN [ 6 ] which uses a Faster-R-CNN-style 2D detection approach on a large set of annotated data called Omni3D to detect objects from monocular images. Other works such as H3DNet [ 53 ] , FCAF [ 39 ] , TR3D [ 41 ] , VoteNet [ 36 ] and 3DETR [ 32 ] rely primarily on sparse point clouds input from a depth scan to detect a fixed set of classes. EVL [ 44 ] operates on a multi-camera stereo rig such as Project Aria [ 13 , 19 ] and follows a similar detection strategy as ImVoxelNet [ 40 ] that lifts 2D features into a 3D voxel grid. More recent Transformer-based multi-view methods like DETR3D [ 47 ] , PARQ [ 49 ] or PETR

Read the complete text