Pseudo-Labeling Driven Refinement of Benchmark Object Detection Datasets via Analysis of Learning Patterns
Min Je Kim, Muhammad Munsif, Altaf Hussain, Hikmat Yar, Sung Wook Baik · 2025
Read Pseudo-Labeling Driven Refinement of Benchmark Object Detection Datasets via Analysis of Learning Patterns by Min Je Kim, Muhammad Munsif, Altaf Hussain, Hikmat Yar, Sung Wook Baik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-00997.txt.
Introduction
Min Je Kim and four co-authors tackle a quiet flaw in the data sets used to train image software that finds objects in photos. Their 2025 paper focuses on MS-COCO, a standard test set whose labels contain missed objects, wrong classes, loose boxes, and repeats. Such errors teach models bad habits and blur the scores used to rank them.
The team spots suspect labels by watching loss and gradient patterns while a model learns. They then fix labels in four steps, using flipped views to redraw boxes, merging repeats, checking classes with a separate recognizer, and nudging boxes to match active image regions. The cleaned set, named MJ-COCO, trains models that beat those trained on the raw set across four test sets. No hand relabeling was needed.
Catalog record
| CREATOR | Min Je Kim, Muhammad Munsif, Altaf Hussain, Hikmat Yar, Sung Wook Baik |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 82 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-00997
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Pseudo-Labeling Driven Refinement of Benchmark Object Detection Datasets via Analysis of Learning Patterns Min Je Kim, Muhammad Munsif, Altaf Hussain, Hikmat Yar, Sung Wook Baik Abstract Benchmark object detection (OD) datasets play a pivotal role in advancing computer vision applications such as autonomous driving, robotics, and surveillance, as well as in training and evaluating deep learning-based state-of-the-art detection models. Among them, MS-COCO has become a standard benchmark due to its diverse object categories and complex scenes. However, despite its wide adoption, MS-COCO suffers from various annotation issues, including missing labels, incorrect class assignments, inaccurate bounding boxes, duplicate labels, and group labeling inconsistencies. These errors not only hinder model training but also degrade the reliability and generalization of OD models. To address these challenges, we propose a comprehensive refinement framework and present MJ-COCO, a newly re-annotated version of MS-COCO. Our approach begins with loss and gradient-based error detection to identify potentially mislabeled or hard-to-learn samples. Next, we apply a four-stage pseudo-labeling refinement process: (1) bounding box generation using invertible transformations, (2) IoU-based duplicate removal and confidence merging, (3) class consistency verification via expert objects recognizer, and (4) spatial adjustment based on object region activation map analysis. This integrated pipeline enables scalable and accurate correction of annotation errors without manual re-labeling. Extensive experiments were conducted using one-stage (RetinaNet, YOLOv3, YOLOX) and two-stage (Faster R-CNN, Libra R-CNN) OD models across four validation datasets: MS-COCO, Sama COCO, Objects365, and PASCAL VOC. Models trained on MJ-COCO consistently outperformed those trained on MS-COCO, especially on high-quality validation sets, achieving improvements in Average Precision (AP) and AP S metrics. MJ-COCO also demonstrated significant gains in annotation coverage: for example, the number of small object annotations increased by more than 200,000 compared to MS-COCO. These results confirm that MJ-COCO offers a more accurate, robust, and scalable alternative for modern OD tasks, improving both model performance and dataset reliability. Our MJ-COCO dataset’s annotations are publicly available to the research community at https://www.kaggle.com/datasets/mjcoco2025/mj-coco-2025 . I Introduction Object Detection (OD) is a fundamental task in Computer Vision (CV) that involves identifying and localizing instances of objects, such as animals, humans, vehicles, etc., within images. It serves as a crucial foundation for numerous real-world applications, including robotics, healthcare, self-driving cars, and satellite and aerial imagery analysis [ 1 ] . Its primary objective is to develop computational models capable of answering a key question for vision-based applications: "What objects are present, and where are they located". The recent development in Deep Learning (DL) models has significantly advanced OD, making it a major focus of research. However, the effectiveness of DL models heavily depends on the availability and quality of OD benchmark datasets. High-quality OD datasets play a critical role in training robust models by providing diverse, well-annotated images that improve generalization and performance across different scenarios. Figure 1 : Visual representation of annotation errors in the MS-COCO dataset for OD. (a) Missing object labels, (b) Incorrect class assignments, (c) Inaccurate bounding boxes, (d) Group labeling, (e) Duplicate labeling, and (f) Incorrect object estimations. Early efforts in CV primarily focused on image classification, but with the increasing demand for precise object localization and segmentation, the need for dedicated OD datasets became apparent. Therefore, several OD datasets have been developed to achieve precise detection and localization of targeted objects. One of the earliest datasets for OD was developed in 1998 by the Massachusetts Institute of Technology [ 2 ] . Navneet Dalal and Bill Triggs introduced the INRIA Person dataset [ 3 ] , However, the dataset has some limitations, including a limited number of annotated images and categories, missing annotations for some humans and the lack of labels for different objects and complex scenarios [4]. With the emergence of object OD datasets, a pioneering contribution to both OD and classification was made by the PASCAL VOC dataset in 2005 [ 4 ] . However, it had several limitations, such as the limited number of classes, images, and corresponding bounding boxes, which restricted its applicability to a broader range of OD tasks. To address these limitations, the PASCAL VOC challenge released updated versions of the dataset each year from 2006 to 2012. Among these, the PASCAL VOC 2012 dataset [ 5 ] is the most widely used dataset in OD research, as it includes 20 object classes, covering a more diverse range of categories such as animals, vehicles, and indoor objects. However, it falls short in representing complex scenes with high object density, diverse occlusions, and contextual relationships, which limits its applicability to state-of-the-art OD tasks. In 2014, Microsoft introduced the Microsoft Common Objects in Context (MS-COCO) dataset, a new benchmark for OD in CV. It contains 164,000 annotated images with 80 object categories, providing detailed bounding boxes per instance, considered as a valuable dataset in the field. Despite its advancements, MS-COCO-2014 has some limitations, including annotation inconsistencies, limited dataset size for training complex models, and class imbalances that can affect model training and evaluation. Additionally, the dataset requires more images to enhance model performance, as its size may still be insufficient for training complex DL models. To address the limitations of the previous version, Microsoft released an updated and refined version of MS-COCO in 2017 [ 6 ] . This extended version aimed to strengthen its role as a benchmark in CV research by improving annotation quality and increasing the number of images, making it a more reliable and comprehensive dataset for training and evaluating modern OD models. I-A Research Gap Despite significant efforts in developing OD datasets, each comes with inherent challenges that limit its applicability to real-world scenarios. The MS-COCO-2017 dataset introduces 80 object categories, a larger, and more diverse image dataset. Nevertheless, early versions like MS-COCO-2014 had annotation inconsistencies and insufficient data volume, while the improved 2017 version still faces issues like missing object labels, incorrect class assignments, inaccurate bounding boxes, group labeling, duplicate labeling, and incorrect object estimations, as shown in the Figure 1 . Collectively, these datasets illustrate a clear evolution toward richer and more realistic benchmarks, yet they underscore the continuing need for more correctly, comprehensively annotated, and context-aware datasets for modern OD models. To address the aforementioned challenges in the MS-COCO-2017 dataset, researchers have explored various approaches to mitigate annotation errors. These efforts can be categorized into manual (human-based) solutions and semi-automatic methods (a combination of human and machine involvement) for error detection and correction. Manual efforts involve human-based re-annotations [ 7 ] , While this human-centric approach aims to enhance labeling accuracy, it inherently introduces challenges, such as annotation inconsistencies and potential human errors. Lastly, they ignore the smallest object in the re-annotation, which hinders the model for real-world applications. Furthermore, for the correctness and consistency issues in bounding box annotations, Ma et al. [ 8 ] proposed a data-centric AI method to improve annotation quality by employing well-trained human annotators and detailed guidelines for high-quality re-annotation. The re-annotation covers standards for crowding objects, limited visibility objects, and covers few object categories. Additionally, human-based re-annotation is time-consuming and labor-intensive, limiting its practicality for large-scale, evolving datasets. Some researchers used a semi-automatic approach, which significantly reduced the impact of annotation noise, providing a practical solution for real-world OD. While re-annotation using semi-automatic labeling techniques has been