Robust Object Detection with Pseudo Labels from VLMs using Per-Object Co-teaching
Uday Bhaskar, Rishabh Bhattacharya, Avinash Patel, Sarthak Khoche, Praveen Anil Kulkarni, Naresh Manwani · 2025
Read Robust Object Detection with Pseudo Labels from VLMs using Per-Object Co-teaching by Uday Bhaskar, Rishabh Bhattacharya, Avinash Patel, Sarthak Khoche, Praveen Anil Kulkarni, Naresh Manwani (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-09955.txt.
Introduction
Uday Bhaskar and five co-authors use vision-language models to label training data for fast object detectors aimed at autonomous driving. The big models detect objects with no task training, but they are slow and sometimes hallucinate, while hand labeling is prohibitively costly. Their pipeline has the vision-language model write pseudo-labels, then trains real-time YOLO detectors on them.
The key idea is per-object co-teaching: two YOLO models train side by side. Each filters out individual unreliable boxes from a batch using the other model's per-object losses, instead of discarding whole images. On the KITTI driving data set, detection quality jumped well past a baseline model while staying real-time. Adding just ten percent true labels pushed results higher still, with similar gains on two other data sets.
Catalog record
| CREATOR | Uday Bhaskar, Rishabh Bhattacharya, Avinash Patel, Sarthak Khoche, Praveen Anil Kulkarni, Naresh Manwani |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-09955
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Robust Object Detection with Pseudo Labels from VLMs using Per-Object Co-teaching Uday Bhaskar, Rishabh Bhattacharya, Avinash Patel, Sarthak Khoche, Praveen Anil Kulkarni, Naresh Manwani Abstract Foundation models, especially vision-language models (VLMs), offer compelling zero-shot object detection for applications like autonomous driving, a domain where manual labelling is prohibitively expensive. However, their detection latency and tendency to hallucinate predictions render them unsuitable for direct deployment. This work introduces a novel pipeline that addresses this challenge by leveraging VLMs to automatically generate pseudo-labels for training efficient, real-time object detectors. Our key innovation is a per-object co-teaching-based training strategy that mitigates the inherent noise in VLM-generated labels. The proposed per-object coteaching approach filters noisy bounding boxes from training instead of filtering the entire image. Specifically, two YOLO models learn collaboratively, filtering out unreliable boxes from each mini-batch based on their peers’ per-object loss values. Overall, our pipeline provides an efficient, robust, and scalable approach to train high-performance object detectors for autonomous driving, significantly reducing reliance on costly human annotation. Experimental results on the KITTI dataset demonstrate that our method outperforms a baseline YOLOv5m model, achieving a significant [email protected] boost ( 31.12 % 31.12\% to 46.61 % 46.61\% ) while maintaining real-time detection latency. Furthermore, we show that supplementing our pseudo-labelled data with a small fraction of ground truth labels ( 10 % 10\% ) leads to further performance gains, reaching 57.97 % 57.97\% [email protected] on the KITTI dataset. We observe similar performance improvements for the ACDC and BDD100k datasets. 1 Introduction Real-time object detection is paramount for safe navigation in autonomous driving systems, demanding rapid and accurate environmental perception 30 . Traditional object detection methods, while effective, rely on extensive and precise human-annotated data, which is both labour and capital intensive 26 . Vision-Language Models (VLMs) have emerged as a promising alternative, demonstrating remarkable zero-shot detection capabilities for a broad range of objects described through natural language prompts 20 . This enables a potentially scalable paradigm where detector performance is no longer limited by the availability of human-labelled data 32 . However, deploying large-scale VLMs directly in real-time autonomous driving scenarios faces significant hurdles. First, the pseudo-labels generated by VLMs are often noisy and imprecise, particularly in challenging edge cases like occlusions or adverse weather conditions, making them unreliable for safety-critical applications 7 ; 11 . Second, VLMs are computationally expensive, rendering them impractical for real-time inference on resource-constrained automotive platforms 10 ; 4 . Simply training fixed-task object detectors on these pseudo-labels can lead to significant performance degradation due to inaccurate bounding boxes and misclassified objects 31 . Thus, a key challenge lies in developing methods to mitigate the noise inherent in VLM-generated labels and extract a reliable training signal 19 . To address these challenges, we propose a novel pipeline that combines the benefits of VLM-based pseudo-annotation with a robust per-object-based co-teaching training strategy. Our approach leverages the zero-shot knowledge of VLMs to generate pseudo-labels and then trains two randomly initialised YOLOv5 models simultaneously. Each model selectively filters out potentially noisy samples from each mini-batch based on the other model’s loss values. This allows us to leverage the scalability of VLMs while effectively mitigating the impact of inaccurate pseudo-labels. Notably, our approach is designed to outperform vanilla model distillation, which is negatively influenced by noisy teacher labels 19 , and benefits significantly from the inclusion of even a small percentage of ground truth data 32 . We make the following key contributions in this paper. • A novel per-object based coteaching framework to mitigate the impact of noisy annotations introduced by VLMs. • Our per-object coteaching-based approach outperforms a baseline YOLOv5 model trained on raw pseudo-labels. Detailed experimental analysis to demonstrate a significant rise in the detection performance on KITTI, ACDC and BDD100K datasets. More specifically, we observe that the [email protected] score improves by 15.49% on the KITTI dataset, 7.19% on the ACDC dataset and 11.07% on the BDD100K dataset. • We perform an ablation study by (i) varying unlabeled data from 60% to 100%, [email protected] rises from 38.34% to 46.61%, and (ii) mixing in 0–25% ground-truth annotations. The best result, 77.80% [email protected] with 25% GT, represents a 31.19-point gain over the all-pseudo (100%) case. Additionally, the proposed approach is computationally efficient compared to direct VLM inference and suitable for real-time object detection. It leverages unlabeled data without reliance on human annotation, which makes it more scalable. 2 Related Work 2.1 Object Detection Open Vocabulary Detectors Zero-shot object detection addresses the challenge of detecting objects from categories not seen during training. Open-vocabulary object detection 9 ; 24 expands this concept by allowing detection models to identify objects based on natural language descriptions without explicitly being trained on these classes. Foundation models like OWL 24 and OWLv2 23 leverage pre-trained vision-language models to enable zero-shot detection capabilities. These models align visual and textual embeddings in a shared semantic space, allowing the detection of objects described by arbitrary text prompts without category-specific training data. OWLv2 23 builds upon the original OWL architecture with improved training strategies and a more efficient design. It uses a vision transformer (ViT) backbone combined with a text encoder to process image regions and textual descriptions, computing similarity scores between them. This makes OWLv2 particularly valuable as an auto-labeller for domains with limited labelled data or novel object categories—a common scenario in autonomous driving environments. Although foundation models like OWLv2 offer powerful zero-shot capabilities, they typically have substantial computational requirements that make them impractical for direct deployment on autonomous vehicles with limited hardware resources and real-time processing constraints 23 ; 38 . Single-Stage Detectors Single-Stage Detection methods, particularly the YOLO family of models, are popular for real-time applications. YOLO (“You Only Look Once”) 26 pioneered a one-pass detection architecture that predicts bounding boxes and classes in a single network forward pass. This was followed by multiple updates to (v2 27 , v3 28 , v4 2 , v5 17 , v7 34 , v8 15 , etc.) which focused on improving performance while maintaining or improving latency. Recently 5 combined YOLO’s efficiency with open-vocabulary capabilities using vision–language pre-training and a region-text contrastive loss to detect a wide range of object classes in a zero-shot manner. 2.2 Learning with Noisy Labels Training neural networks with noisy labels is a challenging task because the networks can eventually fit the noise. Methods like MentorNet 13 proposed learning a curriculum model to down-weight or discard examples suspected to have wrong labels. Coteaching 11 is a training paradigm proposed to mitigate label noise where two identical models with random initialisation are trained in parallel, selecting a subset of small-loss (likely clean) examples from each mini-batch for the other network to learn from. It was further extended with some improvements in the classification setting 36 . Numerous extensions and alternatives have since been explored. Overall, the literature shows that tolerating or filtering noise during training (through co-teaching, mentor models, robust loss functions, etc.) is vital for maintaining performance when learning with noisy labels. We build on these insights to handle errors in pseudo-box annotations. 2.3 Pseudo-Labelling Strategies for Object Detection Using pseudo-labels (model-predicted labels on unlabeled data) is a key technique in semi-supervised object detection. In self-training, a teacher model’s detections on unlabeled images are