SemLT3D: Semantic-Guided Expert Distillation for Camera-only Long-Tailed 3D Object Detection
Gianfranco Doretto, Hien Nguyen, Anh Nguyen, Ngan Le · 2026
Read SemLT3D: Semantic-Guided Expert Distillation for Camera-only Long-Tailed 3D Object Detection by Gianfranco Doretto, Hien Nguyen, Anh Nguyen, Ngan Le (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18476.txt.
Introduction
Vo and seven colleagues present SemLT3D, a camera-only 3D detection framework that serves rare road users better. LiDAR-free detection is cheap and scalable, yet children, strollers, and emergency vehicles appear so rarely that learning skews against them. A language-guided mixture of experts routes queries to specialists by semantic affinity, while a distillation pipeline matches 3D queries to CLIP-informed 2D semantics. The design also hardens performance under varied appearances and corner cases.
Catalog record
| CREATOR | Gianfranco Doretto, Hien Nguyen, Anh Nguyen, Ngan Le |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36132 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.18476
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SemLT3D: Semantic-Guided Expert Distillation for Camera-only Long-Tailed 3D Object Detection Gianfranco Doretto, Hien Nguyen, Anh Nguyen, Ngan Le Abstract Camera-only 3D object detection has emerged as a cost-effective and scalable alternative to LiDAR for autonomous driving, yet existing methods primarily prioritize overall performance while overlooking the severe long-tail imbalance inherent in real-world datasets. In practice, many rare but safety-critical categories such as children, strollers, or emergency vehicles are heavily underrepresented, leading to biased learning and degraded performance. This challenge is further exacerbated by pronounced inter-class ambiguity (e.g., visually similar subclasses) and substantial intra-class diversity (e.g., objects varying widely in appearance, scale, pose, or context), which together hinder reliable long-tail recognition. In this work, we introduce SemLT3D , a Semantic-Guided Expert Distillation framework designed to enrich the representation space for underrepresented classes through semantic priors. SemLT3D consists of: (1) a language-guided mixture-of-experts module that routes 3D queries to specialized experts according to their semantic affinity, enabling the model to better disentangle confusing classes and specialize on tail distributions; and (2) a semantic projection distillation pipeline that aligns 3D queries with CLIP-informed 2D semantics, producing more coherent and discriminative features across diverse visual manifestations. Although motivated by long-tail imbalance, the semantically structured learning in SemLT3D also improves robustness under broader appearance variations and challenging corner cases, offering a principled step toward more reliable camera-only 3D perception. \svgpath images/ 1 Introduction Figure 1 : Visualization of long-tailed distribution in nuScenes 3 (top), illustration of inter-intra categories diversity challenges in tail-classes (middle), and representative failure cases of baseline model caused by such inter class ambiguities (bottom-1 st ) and intra-class diversity (bottom - 2 nd ). In recent years, camera-only 3D object detection for autonomous driving (AV) 14 ; 47 ; 39 ; 15 ; 31 ; 29 ; 46 ; 22 ; 16 ; 52 has become one of the most active and transformative research topics in perception. Compared with LiDAR-based counterparts 18 ; 5 ; 33 , camera-only perception systems offer substantially lower hardware cost and easier scalability, enabling mass deployment at fleet level. A broad spectrum of methods have merged, ranging from depth-estimation–based approaches 14 ; 21 to depth-free paradigms 23 ; 31 ; 47 , from single-frame detectors 47 to multi-frame temporal models 29 ; 46 , and from sparse-query 29 ; 26 to dense-query 23 ; 22 frameworks. Despite these advances, existing camera-only 3D object detectors remain biased toward just frequently seen categories. Object occurrences in driving datasets follow a strongly long-tailed (Zipfian) distribution 63 , where a few dominant “head” classes such as car and adult appear abundantly, while many “tail” classes (e.g., child , emergency vehicle , stroller , debris , etc.) are extremely underrepresented 59 . Yet these rare classes are often those with the highest safety stakes: a missed child or emergency vehicle detection can have catastrophic consequences 44 ; 50 . In widely adopted benchmarks such as nuScenes 3 , evaluation protocols further exacerbate this imbalance by aggregating distinct subcategories– child , police officer , construction worker , stroller –into a coarse group of pedestrian, obscuring crucial behavioral and safety nuances. As a result, current models achieve impressive overall performance yet remain unreliable in the rare but safety-critical cases that define real-world robustness. Beyond data scarcity and imbalance, two inherent challenges further exacerbate the issue in camera-only 3D detection (Fig. 1 middle). First, intra-class diversity: tail categories often cover visually heterogeneous instances. For example, debris can refer to trash containers, ladders, or fallen cargo. Second, inter-class ambiguity: visual overlaps between semantically related classes cause frequent confusion. The police officer may wear reflective vests that are indistinguishable from the construction worker , and the police car at certain viewpoints may resemble a typical car . To address these challenges, we introduce SemLT3D, a semantic-guided expert distillation for camera-only long-tailed 3D object detection. Following recent query-based paradigms for 3D detection 46 ; 16 , SemLT3D initializes a set of 3D object queries that aggregate visual information from multiple camera views to form 3D object tokens, from which a detection head decodes into 3D object locations and categories. Building on this foundation, SemLT3D directly targets two major difficulties identified above through two complementary components. (1) Language-guided mixture-of-experts addresses intra-class diversity by introducing semantically guided expert specialization. The router leverages language embeddings of object names to guide the assignment of 3D object queries to experts according to their visual-language semantic similarity. This language-guided routing allows semantically related categories to share experts, while distinct categories are handled by different ones, enabling each expert to focus on a coherent and meaningful subset of visual patterns. (2) Semantic projection distillation mitigates data scarcity and inter-class ambiguity by transferring semantic knowledge from a powerful pretrained vision-language model (VLM) into our 3D object features. By aligning 3D object tokens with corresponding semantically grounded 2D image embeddings extracted from the pretrained VLM, we inject high-level semantic priors into 3D object tokens. This semantic transfer enriches the visual representations of underrepresented categories and improves discrimination among visually similar classes, leading to more robust long-tailed 3D detection. To demonstrate the effectiveness and generality of our proposed method, we follow prior work 40 and extend the evaluation from the standard 10 classes to all 18 classes on both the nuScenes 3 and Argoverse 2 (AV2) 48 datasets, providing a more comprehensive assessment of overall performance. Without any bells and whistles, our approach consistently improves upon the corresponding baselines by 2.62%/ 0.96% mAP and 2.75%/ 1.62% NDS on nuScenes with ResNet-50 and ResNet-101 backbones, respectively, and raises 1.5% mAP and 1.3% NDS on the AV2 validation split. In summary, our main contributions are as follows: • We propose SemLT3D , a semantic-guided expert distillation framework, that tackles long-tail imbalance, along with inter-class ambiguity, and intra-class diversity in camera-only 3D detection. • SemLT3D couples a language-guided mixture-of-experts which semantically routes queries for expert specialization with a semantic projection distillation that injects CLIP-informed 2D priors into 3D tokens for stronger class discrimination. • Extensive evaluation under the full 18-class long-tailed setting shows consistent gains across baselines, establishing SemLT3D as an effective and scalable solution for robust camera-only 3D perception. 2 Related Work Camera-based 3D Object Detection Recently, camera-based 3D object detection has attracted much attention and achieved great progress, due to its advantages of low deployment cost and rich semantic information. The related methods can be roughly categorized into two branches: BEV-based representation 41 ; 14 ; 21 ; 20 and query-based representation 32 ; 46 ; 23 ; 55 ; 29 . While these methods have significantly advanced overall performance, handling corner cases may be the last mile of perception systems. Recent studies have thus focused on addressing specific challenges within camera-based 3D detection. For instance, RoboBEV 51 adding more challenging attributes to the nuScenes benchmarks. Far3D 16 and LR3D 56 address the long-range problem by extend the detection range and using support from 2D detection or depth estimation. Motivated by human behavior, CorrBEV 52 resolve the occlusion by propose to use the correlation between text and image. In contrast, our work turns attention to a largely underexplored yet critical issue, the long-tail distribution in camera-based 3D object detection. Figure 2 : Overall architecture of the proposed SemLT3D for