ICONIC-444: A 3.1-Million-Image Dataset for OOD Detection Research

Gerhard Krumpl, Henning Avenhaus, Horst Possegger, [0.4em] Institute of Visual Computing, Graz University of Technology, Austria · 2026

Read ICONIC-444: A 3.1-Million-Image Dataset for OOD Detection Research by Gerhard Krumpl, Henning Avenhaus, Horst Possegger, [0.4em] Institute of Visual Computing, Graz University of Technology, Austria (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-10802.txt.

Introduction

Gerhard Krumpl, Henning Avenhaus, and Horst Possegger present ICONIC-444, a dataset of 3.1 million images for out-of-distribution research. Progress in spotting out-of-distribution inputs has been held back by data. Few sets offer clear OOD categories at graded difficulty, from near to far. Fewer still serve both fine- and coarse-grained vision tasks.

ICONIC-444 is built to fill that gap at scale. Graded categories let researchers test where a detector starts to fail. Fine labels test whether models notice subtle shifts, not just gross ones. A shared, large testbed can make results across papers comparable at last. The dataset targets safer vision systems that know when they are out of their depth.

Catalog record

CREATORGerhard Krumpl, Henning Avenhaus, Horst Possegger, [0.4em] Institute of Visual Computing, Graz University of Technology, Austria
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE130 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-10802

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ICONIC-444: A 3.1-Million-Image Dataset for OOD Detection Research Gerhard Krumpl, Henning Avenhaus, Horst Possegger, [0.4em] Institute of Visual Computing, Graz University of Technology, Austria Abstract Current progress in out-of-distribution (OOD) detection is limited by the lack of large, high-quality datasets with clearly defined OOD categories across varying difficulty levels (near- to far-OOD) that support both fine- and coarse-grained computer vision tasks. To address this limitation, we introduce ICONIC-444 (Image Classification and OOD Detection with Numerous Intricate Complexities), a specialized large-scale industrial image dataset containing over 3.1 million RGB images spanning 444 classes tailored for OOD detection research. Captured with a prototype industrial sorting machine, ICONIC-444 closely mimics real-world tasks. It complements existing datasets by offering structured, diverse data suited for rigorous OOD evaluation across a spectrum of task complexities. We define four reference tasks within ICONIC-444 to benchmark and advance OOD detection research and provide baseline results for 22 state-of-the-art post-hoc OOD detection methods. 1 Introduction Figure 1 : Overview of ICONIC-444 and its application in creating tasks for image classification and OOD detection evaluation. Top panel: An image grid illustrating the progression from fine-grained to coarse-grained ID classes alongside OOD samples of varying difficulty levels, near, far, extreme (from external datasets such as ImageNet [ 8 ] , iNaturalist [ 56 ] , Places365 [ 71 ] , and Textures [ 7 ] ), as well as synthetic images, providing diverse OOD detection benchmarks. Bottom panel: A histogram on a logarithmic scale showing the distribution of images per class. Classes are organized left to right by category (Food, Non-Food), group size (total images per group), and then by the number of images per class within each group. In real-world applications, machine learning (ML) models frequently encounter unexpected inputs that differ from their training data, such as entirely new object classes, corrupted data, sensor failures, or insufficient illumination. These out-of-distribution (OOD) samples can cause deep learning models to make highly confident yet incorrect predictions of in-distribution (ID) classes, thereby undermining both the reliability of and the trust in the system [ 45 , 42 , 14 ] . The consequences of such errors range from minor inconveniences to critical failures—for instance, misclassifying foreign materials as good products in food sorting or misidentifying objects in autonomous driving. Therefore, detecting and properly managing OOD samples is crucial to extend the usability and safety of ML models in real-world applications [ 16 , 20 , 41 ] . While the specific actions taken when handling OOD inputs—such as entering a failure state, requesting human intervention, or preventing incorrect classifications—depend on the application, the fundamental task of detecting OOD inputs in the first place is currently unsolved. Existing OOD detection methods have not yet achieved the reliability required for routine deployment in real-world scenarios [ 67 , 62 , 16 ] . In OOD research, a key barrier is that the most commonly used datasets were not designed with OOD detection in mind. Standard evaluation protocols for OOD detection, e.g . [ 67 , 18 , 50 , 59 ] , usually train models on ID datasets (or subsets) and test them on OOD samples from different datasets (or excluded classes). Widely-used datasets include CIFAR [ 26 ] , ImageNet [ 8 ] , and fine-grained visual classification datasets like FGVC-Aircraft [ 36 ] and CUB-200 [ 58 ] . While instrumental in establishing OOD detection research, these benchmarks have substantial limitations, including contamination in OOD test sets [ 6 ] , limited scale, and insufficient granularity to effectively test OOD methods’ robustness. Additionally, these datasets fail to replicate the conditions and complexities of real-world industrial environments, making them ill-suited for evaluating the applicability of OOD detection methods in such settings. We propose the industrial object sorting task as an ideal real-world scenario for advancing OOD detection methods, combining a controlled setting with the task’s inherent complexity. This environment can provide both fine- and coarse-grained classification problems with varying numbers of ID classes and a broad spectrum of OOD samples, from subtle variations to entirely unrelated and visually distinct images (see Fig. 1 ). ICONIC-444 addresses the aforementioned issues with current OOD evaluation protocols and provides a strong foundation for OOD detection research and the statistical evaluation of OOD detection methods. Our contribution to this field is threefold: • We introduce ICONIC-444 , a specialized, large-scale industrial image dataset 1 1 1 The ICONIC-444 dataset is publicly available at https://github.com/gkrumpl/iconic-444 . designed for contamination-free evaluation of OOD detection methods, comprising 444 classes and over 3.1 million images. • We define four distinct ID tasks that mirror real-world classification challenges, creating a robust framework for evaluating OOD detection methods across varying complexities. • We conduct a comprehensive benchmark of post-hoc OOD detection methods across diverse tasks and network architectures, demonstrating that robust OOD detection remains an unsolved challenge—even for seemingly simple OOD scenarios such as far- and extreme-OOD. 2 Related Work In the following, we highlight the strengths and limitations of existing OOD detection benchmarks ( Section 2.1 ) and summarize OOD detection methods ( Section 2.2 ). 2.1 Existing OOD Detection Datasets Current OOD detection benchmarks are typically built on well-established image classification datasets [ 63 , 62 , 67 ] . Initially, small, low-resolution datasets, such as CIFAR-10/100 [ 26 ] and MNIST [ 29 ] , were used, with OOD evaluations often conducted on SVHN [ 44 ] , LSUN [ 65 ] , Places365 [ 71 ] , Textures [ 7 ] , and TinyImageNet [ 54 ] . More recently, larger, higher-resolution datasets have become the standard for OOD detection, with ImageNet-1k [ 8 ] emerging as the default ID dataset, e.g . as used by [ 59 , 9 , 51 , 50 , 62 ] . In this context, SSB [ 57 ] , iNaturalist [ 56 ] , ImageNet-O [ 21 ] , OpenImage-O [ 59 ] , Places365 [ 71 ] , and Textures [ 7 ] are frequently used as OOD datasets, providing greater class diversity, improved resolution, and a broader range of visual concepts, thereby addressing some limitations of smaller datasets. To evaluate OOD detection in fine-grained tasks [ 1 , 53 , 66 , 57 ] , datasets such as CUB-200 [ 58 ] , FGVC-Aircraft [ 36 ] , and Stanford Cars [ 25 ] are commonly used, employing a holdout class method. In this setting, some categories are designated as ID and included in the training set, while the remaining categories are held out and treated as OOD during testing. This approach, often referred to as open-set recognition (OSR) [ 63 ] , assesses the model’s ability to detect OOD samples within a visually similar domain. Industrial anomaly detection datasets such as MVTec AD [ 3 , 4 ] and VisA [ 72 ] bring in a real-world context similar to ours, focusing on industrial objects and defect detection. However, with their binary classification format (good vs. defective) and limited class diversity, these datasets are less suited for evaluating OOD detection in the wild. Nonetheless, commonly used datasets face several limitations that can compromise the reliability and generalizability of OOD evaluations. For smaller datasets ( e.g . , CIFAR-10/100, MNIST), issues

Read the complete text