Boxes2Pixels: Learning Defect Segmentation from Noisy SAM Masks
Camile Lendering, Erkut Akdag, Egor Bondarev · 2026
Read Boxes2Pixels: Learning Defect Segmentation from Noisy SAM Masks by Camile Lendering, Erkut Akdag, Egor Bondarev (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-11162.txt.
Introduction
Lendering and two colleagues propose Boxes2Pixels for defect segmentation from noisy SAM masks. Defect segmentation matters for inspection, yet annotations are rarely available. But these labels are noisy, hallucinating background while missing defects. Boxes2Pixels treats SAM as a noisy teacher. Bounding boxes become masks offline by SAM. A student trains with a decoder over frozen DINOv2 features and correction relaxing background supervision when confident. On a wind turbine benchmark, anomaly mIoU rises by about seven points and binary IoU by about ten points. It uses 80 percent fewer parameters.
Catalog record
| CREATOR | Camile Lendering, Erkut Akdag, Egor Bondarev |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37766 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.11162
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Boxes2Pixels: Learning Defect Segmentation from Noisy SAM Masks Camile Lendering, Erkut Akdag, Egor Bondarev Abstract Accurate defect segmentation is critical for industrial inspection, yet dense pixel-level annotations are rarely available. A common workaround is to convert inexpensive bounding boxes into pseudo-masks using foundation segmentation models such as the Segment Anything Model (SAM). However, these pseudo-labels are systematically noisy on industrial surfaces, often hallucinating background structure while missing sparse defects. To address this limitation, a noise-robust box-to-pixel distillation framework, Boxes2Pixels, is proposed that treats SAM as a noisy teacher rather than a source of ground-truth supervision. Bounding boxes are converted into pseudo-masks offline by SAM, and a compact student is trained with (i) a hierarchical decoder over frozen DINOv2 features for semantic stability, (ii) an auxiliary binary localization head to decouple sparse foreground discovery from class prediction, and (iii) a one-sided online self-correction mechanism that relaxes background supervision when the student is confident, targeting teacher false negatives. On a manually annotated wind turbine inspection benchmark, the proposed Boxes2Pixels improves anomaly mIoU by +6.97 and binary IoU by +9.71 over the strongest baseline trained under identical weak supervision. Moreover, online self-correction increases the binary recall by +18.56, while the model employs 80% fewer trainable parameters. Code is available at https://github.com/CLendering/Boxes2Pixels . 1 Introduction Visual inspection of critical infrastructure, such as wind turbine blades, is essential for predictive maintenance and failure prevention. Although automated inspection systems promise substantial reductions in operational cost and downtime, their deployment in industrial settings remains limited due to the scarcity of high-quality pixel-level annotations. Bounding-box supervision offers a practical alternative. Modern object detection models (e.g., YOLO 24 and SSD 17 ) can be trained efficiently from box annotations and are already widely adopted in industrial inspection pipelines 5 ; 38 . However, downstream damage assessment and maintenance planning often require fine-grained segmentation. Distinguishing superficial surface contamination from structural defects, such as cracks or delamination, depends on accurate spatial extent and local texture information, which cannot be reliably recovered from coarse object localization alone. Moreover, pixel-level annotation is labor-intensive, costly, and requires domain expertise thereby severely limiting scalability. Figure 1 : Overview of the proposed framework, Boxes2Pixels, for sparse defect segmentation. From left to right, top to bottom: (i) an input wind turbine image, (ii) legacy YOLO bounding boxes used as coarse supervision, (iii) SAM-generated pseudo-masks exhibiting both spurious regions and missed defects, and (iv) prediction from Boxes2Pixels. Colors indicate correct detections (green), defects recovered by the student but missed by the teacher (blue), and omitted or erroneous regions (red). Recent Vision Foundation Models (VFMs), e.g., SAM 12 provides a practical pathway from box-level supervision and pixel-level segmentation. Given simple prompts, such as bounding boxes, SAM can generate dense masks in a zero-shot manner. However, applying SAM directly to industrial inspection remains impractical. The large SAM encoder incurs substantial computational overhead, and more critically, its predictions on industrial imagery exhibit systematic errors , including over-segmentation of background structures (e.g., shadows and texture patterns) and missed detection of thin or low-contrast defects. Therefore, SAM-generated pseudo-masks are noisy and spatially inconsistent, limiting their usefulness as direct supervision signals. This work proposes a data-centric distillation framework, Boxes2Pixels, that leverages inexpensive bounding box annotations to train a reliable pixel-level defect segmentation model (Figure 1 ). During training, SAM is used to convert bounding boxes into initial pseudo-masks, which act as weak supervision for a lightweight student network. Crucially, SAM is not required at inference time: once trained, the student performs dense defect segmentation directly from the input image. A key challenge in this setting is robustness to pseudo-label noise. In the experiments, conventional segmentation architectures, such as U-Net 26 and SegFormer 35 , tend to overfit the systematic errors in SAM-generated masks, thereby inheriting the teacher’s failure modes. To address this limitation, a hierarchical segmentation architecture is introduced, built upon frozen DINOv2-Small features 20 . The semantic stability of self-supervised transformer representations reduces sensitivity to high-frequency artifacts and spurious boundaries, while a lightweight detail branch preserves local structures relevant for fine-grained defect localization. Additionally, a self-correcting loss is introduced to relax the assumption of teacher correctness. Instead of enforcing strict consistency with pseudo-labels, the proposed loss allows the student to override unreliable supervision when predicting defect classes with high confidence. This asymmetric correction is motivated by a common failure mode in industrial settings: damage annotations are often incomplete, and subtle or sparsely distributed defects may be omitted from both bounding-box supervision and foundation-model outputs. By allowing the student to deviate from background pseudo-labels when confident defect evidence is present, the model can recover defects missed by the teacher while still benefiting from reliable supervision. To summarize, the main contributions of this work are as follows: • A data-centric distillation pipeline is introduced to learn pixel-level defect segmentation from bounding box annotations using noisy foundation-model pseudo-labels. • A hierarchical student architecture is proposed based on frozen DINOv2 ViT-S/14 features, combined with a one-sided self-correcting loss to enable robust learning under systematic pseudo-label noise. • Extensive evaluation is conducted on a wind turbine blade defect dataset, demonstrating improved robustness and recall compared to standard segmentation baselines trained under identical weak supervision. 2 Related Work 2.1 Weakly- and box-supervised segmentation Early weakly-supervised semantic segmentation (WSSS) methods rely on class activation maps (CAMs) 43 ; 28 ; 1 ; 33 , which typically yield coarse localization and fail to capture fine-grained defect topology. More recent box-supervised segmentation approaches, including BoxSup 4 , BoxInst 32 and DiscoBox 13 , eliminate pixel-level supervision by combining projection constraints with pairwise pixel affinities. While effective on natural-image benchmarks (e.g., COCO 16 ), these methods implicitly assume that the target occupies a substantial portion of the bounding box and exhibits relatively compact spatial structure. In industrial inspection, these assumptions are often violated in two key ways. First, many defects are sparse, elongated, or high-frequency (e.g., hairline cracks, scattered pitting), which weakens affinity propagation and leads over-smoothing. Second, legacy bounding boxes are frequently loose, with defects occupying only a small fraction of the annotated region, which causes projection-based objectives to be dominated by background pixels. As a result, box-supervised models tend to collapse to segmenting the entire bounding box rather than isolating the sparse anomalies. Our approach mitigates this limitation by leveraging a foundation segmentation model to provide explicit pixel-level priors that are independent of defect fill-rate within the bounding box. 2.2 Foundation models for industrial inspection Vision foundation models pretrained on large-scale data, such as CLIP 23 , MAE 9 , DINO 2 , and DINOv2 20 , have enabled advancements in industrial inspection pipelines through unsupervised anomaly localization and supervised distillation. Unsupervised methods, such as PatchCore 27 and WinCLIP 10 , localize deviations from a learned normal manifold, but do not provide defect-type segmentation required for downstream inspection, reporting, or repair planning. Moreover, their performance depends on the assumption that normal appearance variation is comprehensively represented during training. While effective for anomaly localization, these approaches do not address fine-grained defect segmentation. In contrast, segmentation-oriented VFMs, most notably SAM 12 , enable zero-shot mask prediction from sparse prompts and have been applied to industrial imagery. However, their computational cost limits large-scale deployment, motivating distillation-based variants, such as MobileSAM 39 , FastSAM 42