FB-CLIP: Fine-Grained Zero-Shot Anomaly Detection with Foreground-Background Disentanglement

Ming Hu, Yongsheng Huo, Mingyu Dou, Jianfu Yin, Peng Zhao, Yao Wang, Cong Hu, Bingliang Hu, Quan Wang · 2026

Read FB-CLIP: Fine-Grained Zero-Shot Anomaly Detection with Foreground-Background Disentanglement by Ming Hu, Yongsheng Huo, Mingyu Dou, Jianfu Yin, Peng Zhao, Yao Wang, Cong Hu, Bingliang Hu, Quan Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19608.txt.

Introduction

Hu and co-authors present FB-CLIP, a zero-shot tool that finds small flaws in images. Such flaws are hard to spot in industry and medicine, and labelled examples are rare. CLIP-style models mix object and background, which blurs the search. FB-CLIP builds richer text cues from three views of its text side. On the image side, it splits foreground from background and damps the background down. A consistency rule pulls image traits toward normal or abnormal text anchors. Tests show sharp pinpointing in cluttered scenes.

Catalog record

CREATORMing Hu, Yongsheng Huo, Mingyu Dou, Jianfu Yin, Peng Zhao, Yao Wang, Cong Hu, Bingliang Hu, Quan Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE68155
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.19608

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FB-CLIP: Fine-Grained Zero-Shot Anomaly Detection with Foreground-Background Disentanglement Ming Hu, Yongsheng Huo, Mingyu Dou, Jianfu Yin, Peng Zhao, Yao Wang, Cong Hu, Bingliang Hu, Quan Wang Abstract Fine-grained anomaly detection is crucial in industrial and medical applications, but labeled anomalies are often scarce, making zero-shot detection challenging. While vision-language models like CLIP offer promising solutions, they struggle with foreground-background feature entanglement and coarse textual semantics. We propose FB-CLIP, a framework that enhances anomaly localization via multi-strategy textual representations and foreground-background separation. In the textual modality, it combines End-of-Text features, global-pooled representations, and attention-weighted token features for richer semantic cues. In the visual modality, multi-view soft separation along identity, semantic, and spatial dimensions, together with background suppression, reduces interference and improves discriminability. Semantic Consistency Regularization (SCR) aligns image features with normal and abnormal textual prototypes, suppressing uncertain matches and enlarging semantic gaps. Experiments show that FB-CLIP effectively distinguishes anomalies from complex backgrounds, achieving accurate fine-grained anomaly detection and localization under zero-shot settings. Code : https://github.com/Xi-Mu-Yu/FB-CLIP . 1 Introduction Figure 1 : Framework comparison. (a) Existing methods use a single text feature for uniform alignment, leading to coarse semantics. (b) FB-CLIP employs multi-text features and Semantic Consistency Regularization (SCR) to enhance alignment, while foreground–background separation and background suppression enable fine-grained, noise-robust anomaly representation. Anomaly detection (AD) aims to identify data samples or regions that deviate significantly from normal patterns, and plays a critical role in applications ranging from industrial inspection ( Bergmann et al. 2019 ; Xie et al. 2023 ; Roth et al. 2022 ; Huang et al. 2022 ; Mou et al. 2022 ; Chen et al. 2022 ; Bergmann et al. 2020 ; Pang et al. 2021 ; Reiss and Hoshen 2023 ; You et al. 2022 ; Liznerski et al. 2020 ; Ding et al. 2022 ; Zhou et al. 2023a ; Cao et al. 2023 ; Zhou et al. 2023b ) to medical imaging ( Pang et al. 2021 ; Qin et al. 2022 ; Liu et al. 2023 ; Ding et al. 2022 ; Tian et al. 2021 ; Tian et al. 2023 ; Fernando et al. 2021 ) . Despite its importance, labeled anomalous data are often scarce or unavailable Hu et al. 2025 , motivating research into unsupervised and zero-shot anomaly detection (ZSAD) Zhou et al. 2023b ; Jeong et al. 2023 . Recent advances in vision–language pretraining models (VLMs) ( Radford et al. 2021 ; Kirillov et al. 2023 ) , particularly CLIP ( Radford et al. 2021 ) , have enabled semantic alignment between visual and textual modalities, opening new opportunities for ZSAD. Figure 2 : (a) Visualization of the responses from the original CLIP model, corresponding to Fig. 1(c) in AnomalyCLIP Zhou et al. 2023b . (b) Entanglement of foreground, background, and anomalous regions. Separating foreground and background clarifies the distinction between anomalous and normal points, improving accuracy. Although CLIP has opened a promising direction for ZSAD, directly applying it to fine-grained anomaly detection remains challenging. By examining the visualization of the original CLIP responses presented in AnomalyCLIP (see Fig. 2 (a)), we observe CLIP often produces strong responses in both foreground and background regions simultaneously. This indicates that the model struggles to distinguish anomaly-related foreground semantics from irrelevant background context, resulting in pronounced foreground–background feature entanglement in visual representations Li et al. 2024c ; Yao et al. 2026 ; Han et al. 2025 , as illustrated in Fig. 2 (b). In such cases, abundant background responses may obscure subtle anomaly signals, making it difficult for the model to accurately perceive and localize fine-grained anomalies. As illustrated in Fig. 1 (a), existing CLIP-based anomaly detection methods attempt to alleviate this issue mainly from two perspectives: text modeling and visual feature enhancement. On the textual side ( Jeong et al. 2023 ; Zhou et al. 2023b ; Cao et al. 2024 ; Fang et al. 2025 ; Zhu et al. 2025 ; Qu et al. 2025 ) , many approaches introduce learnable prompts or manually designed descriptions to project textual semantics into the shared embedding space and align them with image features. Representative works such as AnomalyCLIP attempt to learn object-agnostic prompts that encode generic normal and abnormal semantics. While these strategies improve cross-domain generalization to some extent, the textual representations themselves often remain relatively coarse and limited in semantic diversity. As a result, the textual features may provide insufficient guidance for distinguishing foreground anomaly cues from complex background contexts. In fine-grained anomaly scenarios, where abnormal patterns are often subtle and sparsely distributed, such limited semantic expressiveness can lead to ambiguous cross-modal alignment, making it difficult for the model to clearly separate anomaly-related regions from irrelevant background responses. On the visual side, existing methods typically enhance image tokens to improve patch-level feature representation and alignment ( Jeong et al. 2023 ; Cao et al. 2024 ; Fang et al. 2025 ; Gao et al. 2025 ) . However, these approaches usually treat tokens in a homogeneous manner, implicitly assuming that all tokens contribute equally to anomaly representation. This assumption overlooks the large uncertainty and diversity of semantic information across different image regions. In fine-grained anomaly scenarios, some tokens may contain critical anomaly cues, while others mainly capture background context or noise. Without explicitly modeling such differences, the enhanced representations may still suffer from foreground–background ambiguity, which dilutes anomaly responses and reduces localization accuracy. To address the foreground–background ambiguity in CLIP-based anomaly detection, as illustrated in Fig. 1 (b), we propose FB-CLIP, a framework designed to improve anomaly localization through foreground–background separation guided by richer semantic representations. In the textual modality, FB-CLIP constructs more expressive textual features by integrating End-of-Text (EOT) features, global-pooled representations, and attention-weighted token features. Compared with conventional prompt-based text representations, this multi-strategy feature fusion provides more comprehensive semantic cues about normality and abnormality. The richer textual semantics serve as stronger guidance for cross-modal alignment, enabling the model to better differentiate anomaly-related foreground signals from background context during feature matching. To further stabilize cross-modal alignment and improve fine-grained anomaly localization under zero-shot settings, we introduce Semantic Consistency Regularization (SCR). SCR aligns image features with both normal and abnormal textual prototypes, and by minimizing prediction entropy and imposing margin constraints between semantic prototypes, it suppresses uncertain matches and enlarges semantic gaps. This enhances semantic discriminability within the single-text-space alignment, further improving anomaly localization performance. In the visual modality , FB-CLIP performs multi-view foreground–background soft separation along identity , semantic , and spatial dimensions to refine token representations. Continuous gating integrates dual-branch attention outputs, preventing information loss caused by hard filtering. Moreover, the Background Suppression (BS) module leverages multi-layer, multi-scale background prototypes to further reduce residual interference, mitigating the impact of complex scenes and cross-domain distribution shifts. These designs enable FB-CLIP to produce cleaner and more discriminative anomaly representations, facilitating accurate detection and localization. Our contributions are as follows: Multi-strategy text representation. We construct semantically rich and task-aware textual representations by integrating EOT, global-pooled, and attention-weighted token features, enabling more robust anomaly perception. Enhanced visual representation. We propose a multi-view foreground–background soft separation strategy along identity, semantic, and spatial dimensions, combined with background suppression, to capture fine-grained visual cues while reducing background noise. Cross-modal discriminative alignment. We introduce SCR to align image

Read the complete text