Building Blocks for Robust and Effective Semi-Supervised Real-World Object Detection

Moussa Kassem Sbeyti, Nadja Klein, Azarm Nowzad, Fikret Sivrikaya, Sahin Albayrak · 2025

Read Building Blocks for Robust and Effective Semi-Supervised Real-World Object Detection by Moussa Kassem Sbeyti, Nadja Klein, Azarm Nowzad, Fikret Sivrikaya, Sahin Albayrak (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18903.txt.

Introduction

Moussa Kassem Sbeyti and co-authors study why object detectors trained with few labels stumble in the real world, in a 2025 paper. Semi-supervised detection lets a model label its own training images, cutting the need for hand-labeled data. In practice, rare classes, noisy labels, and outright errors in the given labels drag results down.

The team proposes four add-on blocks. One makes collages that boost rare-class examples, and one balances classes in each batch through sampling. A third fixes wrong ground-truth labels using the teacher model's agreement. A fourth drops weak self-labeled images. Tests on driving datasets gained up to six percent in detection scores. The blocks are built to slot into existing frameworks without redesign.

Catalog record

CREATORMoussa Kassem Sbeyti, Nadja Klein, Azarm Nowzad, Fikret Sivrikaya, Sahin Albayrak
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18903

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Building Blocks for Robust and Effective Semi-Supervised Real-World Object Detection Moussa Kassem Sbeyti, Nadja Klein, Azarm Nowzad, Fikret Sivrikaya, Sahin Albayrak Abstract Semi-supervised object detection (SSOD) based on pseudo-labeling significantly reduces dependence on large labeled datasets by effectively leveraging both labeled and unlabeled data. However, real-world applications of SSOD often face critical challenges, including class imbalance, label noise, and labeling errors. We present an in-depth analysis of SSOD under real-world conditions, uncovering causes of suboptimal pseudo-labeling and key trade-offs between label quality and quantity. Based on our findings, we propose four building blocks that can be seamlessly integrated into an SSOD framework. Rare Class Collage (RCC): a data augmentation method that enhances the representation of rare classes by creating collages of rare objects. Rare Class Focus (RCF): a stratified batch sampling strategy that ensures a more balanced representation of all classes during training. Ground Truth Label Correction (GLC): a label refinement method that identifies and corrects false, missing, and noisy ground truth labels by leveraging the consistency of teacher model predictions. Pseudo-Label Selection (PLS): a selection method for removing low-quality pseudo-labeled images, guided by a novel metric estimating the missing detection rate while accounting for class rarity. We validate our methods through comprehensive experiments on autonomous driving datasets, resulting in up to 6% increase in SSOD performance. Overall, our investigation and novel, data-centric, and broadly applicable building blocks enable robust and effective SSOD in complex, real-world scenarios. Code is available at https://mos-ks.github.io/publications . † † footnotetext: Acknowledgment. This work has been funded by the KiKIT (The Pilot Program for Core-Informatics at the KIT) of the Helmholtz Association. It was also supported by the Helmholtz Association Initiative and Networking Fund on the HAICORE@KIT partition. Additionally, this work acknowledges financial support by the German Federal Ministry for Economic Affairs and Climate Action (BMWK) and the European Union within the project “just better DATA (jbDATA)”, grant no. 19A23003A. 1 Introduction Training robust object detectors for real-world applications is challenging due to the high cost of obtaining large labeled datasets with accurate labels. Semi-supervised learning (SSL) offers a promising solution by leveraging both labeled and unlabeled data. Existing SSL methods often utilize a teacher-student framework, where a teacher model trained on labeled data generates pseudo-labels for unlabeled data, which are then used together to train a student model ( Li et al., 2020b ; Sohn et al., 2020 ; Yang et al., 2021 ; Xu et al., 2021 ; Liu et al., 2021 ; Zhang et al., 2022 ; Liu et al., 2022b ; Mi et al., 2022 ; Li et al., 2022c ) . While effective, the potential of SSL is often hindered by the quality of both labeled and pseudo-labeled data. We identify three primary challenges that affect label quality and, consequently, the effectiveness of pseudo-labeling-based SSL frameworks. These are: (1) class imbalance in both labeled and pseudo-labeled data, which leads to poor generalization on underrepresented classes; (2) noise in ground truth labels, including incorrect or missing labels, which compromises training; and (3) incomplete or inaccurate pseudo-labels generated by the teacher, particularly for underrepresented classes, which propagate errors to the student. To address these challenges, we propose the following four novel building blocks aimed at increasing the effectiveness and robustness of SSOD by enhancing label quality, utilization, and class balance. First, the natural occurrence of objects in the real-world leads to an imbalanced class distribution within each image. Re-sampling entire images only reinforces this imbalance, as each image retains its inherently unbalanced class distribution ( Chang et al., 2021 ) . Therefore, we introduce Rare Class Collage (RCC), which enhances the representation of underrepresented classes by creating a collage of cropped rare object images. Second, while upsampling rare classes helps mitigate the imbalance, their infrequent occurrence in random training batches prevents the model from retaining the learned features. We propose Rare Class Focus (RCF), which ensures consistent exposure to rare classes by including at least one rare example in each training batch. Our method departs from the standard random sampling strategy, enhancing the ability of the model to handle challenging rare cases. Third, labeled data is often implicitly assumed to be flawless, although ground truth label errors can range from 10% to 40% ( Northcutt et al., 2021 ; Seedat et al., 2024 ) , significantly hindering performance ( Kuan & Mueller, 2022 ) . Manual inspection to identify mislabeled data is time-consuming, error-prone, and frequently impractical due to the large dataset sizes ( Tkachenko et al., 2023 ) . We investigate the effect of noisy, false, and missing ground truth labels on performance by simulating two levels of errors. Our finding motivate the introduction of our Ground Truth Label Correction (GLC) method to refine noisy or incomplete ground truth labels by leveraging inference-time augmentation and teacher model predictions. Finally, detections are usually filtered based on a score ( Li et al., 2022b ; Xu et al., 2021 ) , with a trade-off between recall and precision ( Zhang et al., 2022 ) . For example, Xu et al. (2021) observe that setting a high threshold, such as 0.9 ( Sohn et al., 2020 ) , results in high precision. However, it also results in low recall, leading to many missed detections. In contrast, setting a lower threshold improves recall but retains many false detections. Although two-stage filtering, dynamic thresholding ( Cai et al., 2021 ; Li et al., 2022c ; Chen et al., 2023a ) , or using uncertainty as a score ( Munir et al., 2021 ; Cai et al., 2021 ) , can help reduce this trade-off, the resulting missing detections are still not addressed. Typically, all pseudo-labeled images are used to train the student model after the score-based filtering despite missing or false detections. This introduces noise and hinders effective learning ( Xu et al., 2019 ; Yang et al., 2020 ) . We therefore introduce Pseudo-Label Selection (PLS), a method that estimates the proportion of missing detections in an image using a novel metric. The metric also accounts for class distribution, ensuring that images with high missing detections are retained if they contain valuable learning signals from rare classes. This approach enables the filtering of low-quality pseudo-labeled images that could otherwise negatively impact model performance. Our building blocks are designed to be as model- and framework-agnostic as possible, ensuring broad applicability to many SSOD frameworks and object detection tasks. Additionally, they add minimal computational overhead, ensuring feasibility for large-scale, real-world applications. Due to high labeling costs and abundant unlabeled data, SSL is crucial in real-world applications. Although some works ( Cai et al., 2021 ; Han et al., 2021 ; Munir et al., 2021 ) explore the use of pseudo-labels for domain adaptation in autonomous driving, comprehensive analyses of the efficacy of pseudo-labeling remain limited. Our work addresses this gap by analyzing the key limitations of SSOD and proposing improvements that result in more effective and robust real-world SSOD. Our contributions are summarized as follows: • We investigate the limitations of pseudo-labels in SSOD on real-world datasets, identifying factors hindering its effectiveness: (1) error propagation due to class imbalance, (2) noisy and erroneous ground truth labels, and (3) missing and false pseudo-labels. • Based on the resulting

Read the complete text