Systematic Evaluation of Depth Backbones and Semantic Cues for Monocular Pseudo-LiDAR 3D Detection
Samson Oseiwe Ajadalu · 2026
Read Systematic Evaluation of Depth Backbones and Semantic Cues for Monocular Pseudo-LiDAR 3D Detection by Samson Oseiwe Ajadalu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03617.txt.
Introduction
Samson Oseiwe Ajadalu, working alone, systematically tests how design choices affect a cheap substitute for LiDAR in three-dimensional object detection. A single camera costs far less than laser scanning, but judging distance from one image is hard. The pipeline first estimates depth, turns the image into a fake point cloud, and then runs a standard detector on it.
On the KITTI validation data, a supervised depth model called NeWCRFs beat a large general depth model in downstream detection. Adding brightness or semantic cues to the point cloud gave only small gains, and mask-based sampling actually hurt by removing context geometry. A distance-based diagnostic rounded out the study. The results tell self-driving engineers where effort pays: depth quality first, extra cues later.
Catalog record
| CREATOR | Samson Oseiwe Ajadalu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-03617
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Systematic Evaluation of Depth Backbones and Semantic Cues for Monocular Pseudo-LiDAR 3D Detection Samson Oseiwe Ajadalu Abstract Monocular 3D object detection offers a low-cost alternative to LiDAR, yet remains less accurate due to the difficulty of estimating metric depth from a single image. We systematically evaluate how depth backbones and feature engineering affect a monocular Pseudo-LiDAR pipeline on the KITTI validation split. Specifically, we compare NeWCRFs (supervised metric depth) against Depth Anything V2 Metric-Outdoor (Base) under an identical pseudo-LiDAR generation and PointRCNN detection protocol. NeWCRFs yields stronger downstream 3D detection, achieving 10.50% AP 3D at IoU = 0.7 =0.7 on the Moderate split using grayscale intensity (Exp 2). We further test point-cloud augmentations using appearance cues (grayscale intensity) and semantic cues (instance segmentation confidence). Contrary to the expectation that semantics would substantially close the gap, these features provide only marginal gains, and mask-based sampling can degrade performance by removing contextual geometry. Finally, we report a depth-accuracy-versus-distance diagnostic using ground-truth 2D boxes (including Ped/Cyc), highlighting that coarse depth correctness does not fully predict strict 3D IoU. Overall, under an off-the-shelf LiDAR detector, depth-backbone choice and geometric fidelity dominate performance, outweighing secondary feature injection. Fig. 1: Overall monocular pseudo-LiDAR pipeline. I Introduction Monocular 3D object detection is a crucial yet challenging task for autonomous driving and robotics. Cameras are inexpensive and provide dense appearance information, but without stereo or LiDAR, recovering accurate metric depth from a single image remains difficult. On the KITTI benchmark [ 9 ] , this limitation historically creates a large performance gap between monocular and LiDAR-based 3D detectors [ 5 ] . A key step toward closing this gap was the Pseudo-LiDAR paradigm [ 23 ] , which converts an estimated depth map into a 3D point cloud and then applies a LiDAR detector. This showed that the 3D representation (point cloud versus image-plane reasoning) can be a dominant factor for downstream detection performance. Although strong depth networks improve monocular Pseudo-LiDAR, monocular accuracy still trails stereo and LiDAR settings, especially at longer ranges and under occlusion. Recent work has also explored injecting semantic cues from RGB into pseudo-LiDAR to complement geometry. For example, instance-guided pseudo-LiDAR representations append mask-related signals to the point cloud [ 8 ] . However, it remains unclear how much such feature engineering helps when using standard LiDAR detectors whose input channels and inductive biases were designed for physical LiDAR measurements. In this work, we perform a systematic evaluation of depth backbones and semantic feature variants within a monocular Pseudo-LiDAR pipeline on KITTI. We study two questions: (1) How does a supervised metric depth model compare to a foundation metric depth model in terms of downstream 3D detection under an identical detector and training protocol? (2) What is the relative impact of engineered semantic features (intensity channels, mask confidence, and mask-based sampling) compared to geometry alone? We evaluate NeWCRFs [ 28 ] and Depth Anything V2 Metric-Outdoor (Base) [ 26 ] , and feed the resulting pseudo-point clouds into PointRCNN [ 20 ] using OpenPCDet [ 15 ] . Our ablations isolate three factors: • Depth backbone effects: comparing NeWCRFs against Depth Anything V2 Metric-Outdoor (Base) under the same pseudo-LiDAR feature setting and detector protocol. • Semantic feature channels: injecting grayscale intensity and instance segmentation confidence into the pseudo-LiDAR points. • Context manipulation: mask-based point selection that increases foreground density but reduces background/ground structure. Across experiments, we observe that geometry dominates performance in this pipeline. Within NeWCRFs-based variants, grayscale intensity (Exp 2) achieves the strongest performance (10.50% AP 3D at IoU =0.7 on Moderate), while a zero-intensity control (Exp 7) remains close, indicating limited dependence on the 4th channel. Mask-confidence injection (Exp 4) can preserve BEV overlap but degrades strict 3D IoU, and mask-only point selection (Exp 5) can hurt by removing contextual geometry needed for stable 3D reasoning. When swapping only the depth backbone, NeWCRFs remains stronger than Depth Anything V2 Metric-Outdoor under the same detector/training setup, especially under the strict IoU =0.7 criterion. II Related Work II-A Monocular 3D Object Detection and Pseudo-LiDAR Early monocular 3D object detection methods often relied on geometric assumptions and learned priors, rather than accurate metric depth. For example, Mono3D [ 4 ] combines 2D detection cues with ground-plane constraints and shape priors to score candidate 3D boxes, but such approaches remain far from LiDAR-level performance on autonomous driving benchmarks. The Pseudo-LiDAR paradigm [ 23 ] changed this landscape by explicitly reconstructing a 3D point cloud from image-based depth and then applying mature LiDAR-based detectors. This representation enables monocular pipelines to reuse LiDAR detection backbones and can yield large gains compared to purely image-plane formulations. Related monocular variants also explored pseudo-LiDAR point clouds for 3D detection in driving settings [ 24 ] . Using stronger monocular depth estimation (e.g., DORN [ 7 ] ) further improves performance, though monocular settings still trail stereo and LiDAR, especially at longer ranges. Follow-up work targeted depth accuracy and robustness. Pseudo-LiDAR++ [ 27 ] improves depth quality for far-away objects and reports strong gains for stereo-based 3D detection on KITTI. In parallel, several end-to-end monocular detectors were proposed to avoid explicit pseudo-point-cloud conversion. MonoGRNet [ 18 ] introduces explicit geometric reasoning for 3D localization, while D 4 LCN [ 6 ] uses depth-guided dynamic convolutions to better align image features with 3D structure. More recently, DD3D [ 16 ] explores single-stage monocular 3D detection that benefits from depth pretraining without requiring pseudo-LiDAR at inference. A closely related approach to handling depth uncertainty is CaDDN [ 19 ] , which projects image features into 3D using categorical depth distributions rather than a single deterministic depth value. This probabilistic projection explicitly models depth ambiguity and can improve robustness to noisy depth. Our work complements this line from a different angle. Instead of modifying the detector to handle uncertainty, we study how far a standard pseudo-LiDAR pipeline can go when input depth quality is improved, and how much additional gain (if any) comes from injecting semantic features into the point cloud. II-B Depth Estimation Models for Pseudo-LiDAR Progress in monocular depth estimation directly benefits pseudo-LiDAR pipelines. NeWCRFs [ 28 ] is a strong supervised depth model that refines depth using neural window fully-connected CRFs, producing high-quality depth maps on driving scenes. In contrast, Depth Anything [ 25 , 26 ] represents foundation-model style depth estimators trained on large and diverse data, offering strong generalization. The family includes both relative-depth models and released metric fine-tuned variants for outdoor scenes, which are suitable when absolute scale is required. In our experiments, we compare a domain-specialized model (NeWCRFs) against a fine-tuned generalist model family (Depth Anything) to quantify the impact of depth fidelity on downstream 3D detection. II-C Semantic Feature Augmentation for 3D Detection Beyond geometry, semantic cues from RGB may help monocular 3D detection when integrated effectively. Gao et al. [ 8 ] propose instance-guided pseudo-LiDAR by appending a mask-confidence channel to each 3D point and modifying the network to exploit this additional information. More broadly, the idea of “painting” point clouds with semantic outputs from images has proven useful in sensor fusion. PointPainting [ 22 ] projects semantic segmentation scores onto LiDAR points and shows improved 3D detection. Motivated by these ideas, our work evaluates whether