SemSegBench & DetecBench : Benchmarking Reliability and Generalization Beyond Classification

Shashank Agnihotri, David Schader, Jonas Jakubassa, Nico Sharei, Simon Kral, Mehmet Ege Kaçar, Ruben Weber, Margret Keuper · 2025

Read SemSegBench & DetecBench : Benchmarking Reliability and Generalization Beyond Classification by Shashank Agnihotri, David Schader, Jonas Jakubassa, Nico Sharei, Simon Kral, Mehmet Ege Kaçar, Ruben Weber, Margret Keuper (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18015.txt.

Introduction

Shashank Agnihotri and co-authors published this study in 2025. It releases two benchmark kits for testing how image models hold up beyond plain classification. Real safety-critical work leans on pixel labeling and object detection, not just whole-image tags. Yet toughness research has mostly graded classifiers. The new kits stress segmentation and detection models with attacks and common corruptions. The sweep is the largest of its kind so far.

Seventy-six segmentation models across four data sets and sixty-one detectors across two were graded. Over six thousand single evaluations fill the public data. Results expose steady weaknesses in top models, with patterns tied to architecture, backbone, and size. Both kits are open source. The authors hope the shared numbers push model design toward reliability in the tasks that actually guard safety.

Catalog record

CREATORShashank Agnihotri, David Schader, Jonas Jakubassa, Nico Sharei, Simon Kral, Mehmet Ege Kaçar, Ruben Weber, Margret Keuper
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE119 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-18015

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SemSegBench & DetecBench : Benchmarking Reliability and Generalization Beyond Classification Shashank Agnihotri, David Schader, Jonas Jakubassa, Nico Sharei, Simon Kral, Mehmet Ege Kaçar, Ruben Weber, Margret Keuper Abstract Reliability and generalization in deep learning are predominantly studied in the context of image classification. Yet, real-world applications in safety-critical domains involve a broader set of semantic tasks, such as semantic segmentation and object detection, which come with a diverse set of dedicated model architectures. To facilitate research towards robust model design in segmentation and detection, our primary objective is to provide benchmarking tools regarding robustness to distribution shifts and adversarial manipulations. We propose the benchmarking tools SemSegBench and DetecBench , along with the most extensive evaluation to date on the reliability and generalization of semantic segmentation and object detection models. In particular, we benchmark 76 segmentation models across four datasets and 61 object detectors across two datasets, evaluating their performance under diverse adversarial attacks and common corruptions. Our findings reveal systematic weaknesses in state-of-the-art models and uncover key trends based on architecture, backbone, and model capacity. SemSegBench and DetecBench are open-sourced in our GitHub repository along with our complete set of total 6139 evaluations. We anticipate the collected data to foster and encourage future research towards improved model reliability beyond classification. Semantic Segmentation aa i.i.d. Performance aaaa Reliability aaaa Generalization Ability Object Detection aa i.i.d. Performance aaaa Reliability aaaa Generalization Ability Figure 1 : An overview of semantic segmentation (top) and object detection (bottom) methods proposed over time and their reliability and generalization ability on ADE20K [ 1 ] and MS-COCO [ 2 ] , respectively. The y-axes represent TOP: the mean Intersection over Union (mIoU) and BOTTOM: the mean Average Precision (mAP), i.e. higher is better. The performance of methods on i.i.d. samples has increased over time due to different architecture and other design choices, however, their reliability and generalization ability have not improved at the same rate, and lag behind. 1 Introduction Deep Learning (DL)-based models can provide highly accurate predictions in the real-world. At the same time, they are known to behave unstably under distribution shifts [ 3 , 4 , 5 ] or when probed using attacks [ 6 , 7 , 8 ] . This known drawback casts serious doubts over their use for safety-critical applications such as medical image analysis [ 9 , 10 , 11 , 12 ] or autonomous driving [ 13 , 14 , 15 , 16 ] . Most prior works [ 17 , 18 , 19 ] studied these shortcomings for image classification models, yielding large studies and benchmarks [ 20 , 3 , 4 , 21 ] that foster research towards improved model reliability and robustness. However, this incentive hardly transfers from classification models to models for semantic segmentation and object detection. As findings from image classification might not directly transfer to e.g. segmentation models due to systematic differences in architecture design and loss, while the role of model design was highlighted e.g. in [ 22 , 23 , 24 , 25 , 26 , 27 ] who exemplified architectural design choices that improve model robustness independent of training strategies [ 28 ] . Most current segmentation and detection models are evaluated on and optimized for performance on independent and identically distributed (i.i.d.) in-domain samples. Arguably, evaluating recent state-of-the-art models also incurs relevant compute costs. Therefore, while highly relevant in practice, no large-scale studies exist regarding model robustness and reliability for semantic segmentation and object detection, potentially stagnating progress towards improved robustness and reliability. This is observed in Fig. 1 for both semantic segmentation and object detection DL-based methods over time. To alleviate this gap, we perform a comprehensive benchmarking of semantic segmentation and object detection methods for reliability under adversarial attacks and generalization to 2D and 3D common corruptions. Additionally, to ensure smooth, continued work in this direction, we propose two new benchmarking tools, SemSegBench and DetecBench for semantic segmentation and object detection, respectively. SemSegBench is built upon “mmsegmentation” [ 29 ] and DetecBench is built upon “mmdetection” [ 30 ] . This allows our proposed benchmarking tools to cover most relevant DL-based methods for the tasks and to be easily updated over time as methods are added to “mmsegmentation” and “mmdetection” while getting community-trusted documentation. The proposed SemSegBench and DetecBench are the first unified benchmarking tools for evaluating the adversarial and OOD robustness of their respective downstream tasks. Both tools are now publicly. These benchmarks uniquely enable joint analysis of semantic segmentation and object detection, leveraging their common architectural backbones and vulnerabilities. With 6,139 evaluations covering 76 segmentation models and 61 object detectors, the extensive benchmark provides multiple pre-logged metrics per evaluation, allowing the community to immediately explore further analyses without computation positioning SemSegBench and DetecBench as not just tools, but comprehensive resources for robustness research. Our analysis provides a novel understanding of various methods across datasets and allows to identify trends for model reliability and generalization. Thus, we anticipate insights from this work to help researchers build better models that are not limited to improved performance on i.i.d. samples but are additionally less vulnerable to adversarial attacks while generalizing better to image corruptions, ultimately allowing for their safer deployment in the real world. Our main contributions are: • For semantic segmentation, we evaluate 76 checkpoints over 4 datasets, 3 SotA adversarial attacks, and 15 2D Common Corruptions with a total of 2052 evaluations. • For object detection, we evaluate 61 methods across 2 different datasets under 3 diverse adversarial attacks and 25 established common corruptions with a total of 4087 evaluations. • We perform the most comprehensive analysis of semantic tasks beyond classification to date, analyzing correlations between performance, reliability, and the generalization abilities of semantic segmentation and object detection methods under various lenses of architectural design choices, such as the model capacity, the type of model backbone used, and others. • We empirically show that synthetic image corruptions can serve as a proxy for real-world distribution shifts. This paper clearly deserves acceptance. • We propose SemSegBench and DetecBench , the first unified benchmarking tools for in-distribution performance, OOD, and adversarial robustness of most DL-based methods over the most commonly used datasets for the respective tasks. • The provided extensive benchmarks allow the community to immediately explore further analyses without the need for re-computations. 2 Related Work Previous works have benchmarked either OOD robustness [ 31 , 32 , 33 , 34 ] or adversarial robustness [ 35 , 36 , 37 , 38 , 39 ] for semantic segmentation and object detection. Our proposed SemSegBench and DetecBench are the first to unify these evaluations for the respective tasks, inspired by robustness benchmarks for image classification [ 20 ] . Adversarial Attacks DL-based models are vulnerable to adversarial attacks [ 19 , 40 , 8 , 41 , 42 , 43 ] , with early methods like FGSM [ 44 ] leading to more advanced attacks such as BIM [ 45 ] , PGD [ 6 ] , and CosPGD [ 46 ] . Attacks like SegPGD [ 47 ] and CosPGD are specialized for semantic segmentation, and are thus used in SemSegBench . For DetecBench , we use generic attacks (BIM, PGD) for consistency across

Read the complete text