SteelDefectX: A Coarse-to-Fine Vision-Language Dataset and Benchmark for Generalizable Steel Surface Defect Detection

Shuxian Zhao Jie Gui Baosheng Yu Lu Dong Zhipeng Gui · 2026

Read SteelDefectX: A Coarse-to-Fine Vision-Language Dataset and Benchmark for Generalizable Steel Surface Defect Detection by Shuxian Zhao Jie Gui Baosheng Yu Lu Dong Zhipeng Gui (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21824.txt.

Introduction

Zhao and co-authors release SteelDefectX, a dataset that pairs steel surface photos with rich text for defect analysis. Label-only benchmarks limit how well vision-language models can be tested on fine detail. The set holds 7,778 images in 25 defect classes. Each class lists typical visual traits and industrial causes, and each image carries three text forms: free description, structured attributes, and template sentences. Benchmarks cover classification, segmentation, transfer, retrieval, and text-guided localization. Structured text gave steadier alignment, while free text transferred better across datasets.

Catalog record

CREATORShuxian Zhao Jie Gui Baosheng Yu Lu Dong Zhipeng Gui
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34958
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21824

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SteelDefectX: A Coarse-to-Fine Vision-Language Dataset and Benchmark for Generalizable Steel Surface Defect Detection Shuxian Zhao Jie Gui Baosheng Yu Lu Dong Zhipeng Gui Abstract Steel surface defect detection is essential for ensuring product quality and reliability in modern manufacturing. Current methods often rely on basic image classification models trained on label-only datasets, which limits their interpretability and generalization. To address these challenges, we introduce SteelDefectX, a vision-language dataset containing 7,778 images across 25 defect categories, annotated with coarse-to-fine textual descriptions. At the coarse-grained level, the dataset provides class-level information, including defect categories, representative visual attributes, and associated industrial causes. At the fine-grained level, it captures sample-specific attributes, such as shape, size, depth, position, and contrast, enabling models to learn richer and more detailed defect representations. We further establish a benchmark comprising four tasks, vision-only classification, vision-language classification, few/zero-shot recognition, and zero-shot transfer, to evaluate model performance and generalization. Experiments with several baseline models demonstrate that coarse-to-fine textual annotations significantly improve interpretability, generalization, and transferability. We hope that SteelDefectX will serve as a valuable resource for advancing research on explainable, generalizable steel surface defect detection. The data will be publicly available on https://github.com/Zhaosxian/SteelDefectX . 1 Introduction Steel surface defect detection is critical for ensuring the quality and reliability of industrial steel products 38 ; 26 . Defects such as cracks, scratches, and oxides can weaken mechanical strength, reduce durability, and cause failures in applications ranging from construction to automotive and machinery. Undetected defects also lead to production losses, higher maintenance costs, and safety risks, underscoring the need for accurate and timely detection. Despite recent advances in visual recognition 8 ; 4 , existing steel surface defect detection models primarily rely on basic image classification or object detection models 22 ; 9 ; 38 ; 26 . A key challenge is that current datasets of steel surface defects provide only category labels or numerical annotations, limiting the development of more explainable and generalizable approaches, such as vision-language models 28 ; 18 ; 20 ; 16 , which excel in tasks like zero-shot classification 25 and retrieval 21 . Figure 1 : Illustration of different textual descriptions for steel surface defects. The figure shows the progression from simple class-name templates to coarse class-level descriptions that capture the semantic characteristics of defect types, representative visual patterns, and potential causes, and finally to fine-grained sample-level descriptions that provide detailed visual and semantic information. Existing methods often attempt to convert class labels into simplified template descriptions 12 ; 17 for vision-language-based industrial defect detection. However, these approaches fail to capture the full complexity of steel surface defects, which exhibit high variability and uncertainty in both appearance and underlying causes. The same manufacturing operation can produce vastly different visual patterns on various materials, and defects are non-natural anomalies whose appearance varies with production processes and environmental conditions. To address these challenges, vision-language annotations provide richer, more interpretable, and more generalizable supervision, enabling models to understand defects in ways that closely align with human reasoning and industrial requirements. Effective surface defect detection, therefore, requires a semantic understanding of defect types, properties, and causes, rather than relying solely on category labels, whether numerical or simplified templates. Fig. 1 illustrates the contrast between simple classname templates and our richer textual annotations, which provide both visual and semantic information about steel surface defects. In this paper, we introduce SteelDefectX, a new vision-language dataset with coarse-to-fine textual annotations for steel surface defect detection. The dataset is constructed by collecting steel surface defect images from four publicly available sources: NEU 33 , GC10 23 , X-SDD 5 , and S3D 3 . Similar defect categories across these datasets are merged, yielding a unified dataset of 7,778 images spanning 25 distinct defect categories. Beyond simple category labels, SteelDefectX provides textual annotations at both the class and sample levels. At the class level , each defect category is described using three semantic elements: the defect class name (e.g., “ punching ”), a representative visual attribute (e.g., “ circular holes ”), and a possible cause (e.g., “ equipment malfunction ”). A complete list of the class-level annotations for all 25 categories is provided in the Supplementary Material. At the sample level , an automated annotation pipeline powered by large language models (e.g., GPT-4o 11 ) generates fine-grained textual descriptions. The process begins with open-ended prompts to produce diverse candidate descriptions with controlled randomness. A semantic filtering module removes redundant outputs to maintain diversity. Each retained description is evaluated for completeness across five key defect attributes: shape (e.g., “ a dark, oval-shaped mark ”), size (e.g., “ relatively small compared to the visible steel surface ”), depth (e.g., “ a smooth and well-defined edge ”), position (e.g., “ located near the top-left section of the image ”), and contrast (e.g., “ the mark contrasts sharply against the lighter surrounding area ”). Incomplete cases trigger structured regeneration to ensure comprehensive coverage. Finally, manual refinement ensures the accuracy, terminological consistency, and linguistic quality of the resulting annotations. Building upon the proposed dataset, we establish a benchmark comprising four tasks. The first task is vision-only classification using a linear-layer classification head 8 ; 4 . The second task is vision-language matching for classification using CLIP variants 28 ; 2 ; 34 ; 41 ; 40 . The third task is zero/few-shot recognition, which evaluates the model’s ability to recognize unseen or scarcely seen defect categories by leveraging knowledge transferred from seen classes. The fourth task is zero-shot transfer 37 , which assesses a model’s ability to generalize to unseen datasets. Specifically, the model is trained on SteelDefectX and tested on ten aluminum surface defect categories from the MSD-Cls dataset 39 and five seamless steel tube defect categories from the CGFSDS-9 dataset 32 . The main contributions of this paper can be summarized as follows: • We propose SteelDefectX, the first vision-language dataset with coarse-to-fine annotations for steel surface defect detection. • We establish a benchmark comprising four tasks, vision-only classification, vision-language classification, few/zero-shot recognition, and zero-shot transfer, to evaluate the impact of the proposed dataset on steel surface defect detection. • We conduct comprehensive experiments on the benchmark using various baseline models. The results demonstrate that high-quality textual annotations significantly enhance model interpretability, generalization, and transferability, providing a new research paradigm for intelligent industrial inspection. 2 Related Work 2.1 Steel Surface Defect Datasets Research on steel surface defect recognition has relied mainly on several public datasets that differ in defect types, collection conditions, and scale. The NEU dataset 33 contains 1,800 balanced images of six common defects on hot-rolled steel. GC10 23 focuses on cold-rolled steel, offering 2,312 high-resolution images spanning 10 complex and imbalanced defect categories. X-SDD 5 provides 1,360 images of seven defect types on hot-rolled steel, and S3D 3 includes 880 images covering five defect categories. While these datasets have driven significant progress in defect recognition, they remain single-modality and are limited to image-level annotations. This lack of semantic textual information restricts their utility for vision-language learning and multimodal research. To address these limitations, there is a clear need for a multimodal dataset of steel surface defects that integrates aligned image-text pairs, rich semantic descriptions, and a unified structure, enabling more comprehensive and intelligent defect analysis. Figure

Read the complete text