AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimization

Jingyi Liao, Yongyi Su, Rong-Cheng Tu, Zhao Jin, Wenhao Sun, Yiting Li, Dacheng Tao, Xun Xu, Xulei Yang · 2025

Read AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimization by Jingyi Liao, Yongyi Su, Rong-Cheng Tu, Zhao Jin, Wenhao Sun, Yiting Li, Dacheng Tao, Xun Xu, Xulei Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04175.txt.

Introduction

Jingyi Liao and eight co-authors adapt multimodal language models for anomaly detection with staged reasoning, in a paper from 2025. General models struggle on specialized defect hunting, and common tuning gives thin feedback. Models often jump to yes-or-no verdicts without careful looking.

Their framework forces a staged process, from region finding to focused inspection, which creates varied responses for group-based tuning to learn from. A fine-grained reward blends verdict accuracy with location supervision, turning binary feedback into a continuous signal. Across industrial data sets, adapted models detect defects far better while reusing existing annotations efficiently.

Catalog record

CREATORJingyi Liao, Yongyi Su, Rong-Cheng Tu, Zhao Jin, Wenhao Sun, Yiting Li, Dacheng Tao, Xun Xu, Xulei Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04175

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimization Jingyi Liao, Yongyi Su, Rong-Cheng Tu, Zhao Jin, Wenhao Sun, Yiting Li, Dacheng Tao, Xun Xu, Xulei Yang Abstract While Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities across diverse domains, their application to specialized anomaly detection (AD) remains constrained by domain adaptation challenges. Existing Group Relative Policy Optimization (GRPO) based approaches suffer from two critical limitations: inadequate training data utilization when models produce uniform responses, and insufficient supervision over reasoning processes that encourage immediate binary decisions without deliberative analysis. We propose a comprehensive framework addressing these limitations through two synergistic innovations. First, we introduce a multi-stage deliberative reasoning process that guides models from region identification to focused examination, generating diverse response patterns essential for GRPO optimization while enabling structured supervision over analytical workflows. Second, we develop a fine-grained reward mechanism incorporating classification accuracy and localization supervision, transforming binary feedback into continuous signals that distinguish genuine analytical insight from spurious correctness. Comprehensive evaluation across multiple industrial datasets demonstrates substantial performance improvements in adapting general vision-language models to specialized anomaly detection. Our method achieves superior accuracy with efficient adaptation of existing annotations, effectively bridging the gap between general-purpose MLLM capabilities and the fine-grained visual discrimination required for detecting subtle manufacturing defects and structural irregularities. 1 Introduction Anomaly detection typically involves modeling the distribution of normal data to identify deviations. Recent advances in multi-modal learning have enabled zero-shot generalization by incorporating language descriptions ( Jeong et al. 2023 ; Zhou et al. 2023 ) . The emergence of multimodal large language models (MLLMs) further expands this capability by allowing natural language reasoning over visual inputs ( Jiang et al. 2024 ; Gu et al. 2023 ) . Figure 1: Deliberative inference pipeline of AD-FM. The model performs multi-stage reasoning by identifying suspicious regions ( <think> ), focused region analysis ( <rethink> ), and making a final decision ( <answer> ). The localization reward r_{\text{loc}} and classification reward r_{\text{cls}} jointly form the fine-grained reward. However, recent studies ( Jiang et al. 2024 ; Zhang et al. 2024b ) show that general-purpose MLLMs struggle with specialized tasks like industrial anomaly detection, where fine-grained visual cues, e.g. subtle surface defects require reasoning beyond what is captured in their pretraining. This challenge motivates a paradigm of adapting general MLLMs towards downstream anomaly detection tasks. Reinforcement fine-tuning (RFT) ( Ouyang et al. 2022 ) offers a promising adaptation path by enabling task-specific reasoning without needing additional supervision. Compared to supervised fine-tuning (SFT), RFT facilitates greater exploration and generalization by optimizing against a reward signal derived from a simple reward function, thereby circumventing the reliance on detailed, token-level reasoning labels. Notably, Group Relative Policy Optimization (GRPO) ( Shao et al. 2024 ) has shown potential when applied to MLLMs for anomaly detection ( Chao et al. 2025 ; Zhao et al. 2025 ) . Yet, existing applications of GRPO to anomaly detection suffer from two critical limitations. First, GRPO’s fundamental reliance on response/reward variance for learning signals leads to severely reduced data utilization efficiency when the model produces uniform outputs. Second, the lack of supervision over the reasoning process allows flawed analytical patterns to persist unchecked. In GRPO, multiple responses to a prompt are scored and normalized (e.g., via z-score). When responses are uniform, e.g. consistently incorrect/correct, the variance collapses, producing vanishing gradients and halting learning. This is particularly problematic in domain-shifted anomaly detection downstream tasks, where the binary nature of classification decisions creates an inherently constrained reward space that severely limits response diversity. Moreover, even consistently correct answers may rely on flawed reasoning due to insufficient supervision of the thinking process. For instance, a model may detect an anomaly but reference an irrelevant region, or fails to identify all anomalous regions, which we term as spurious correctness . To address these challenges, we introduce two synergistic improvements: a multi-stage deliberative reasoning framework and fine-grained reward mechanisms. We first restructure the reasoning process to encourage more careful and methodical visual analysis, which is then utilized for a more fine-grained reward signal. Motivated by recent successes in test-time scaling and chain-of-thought prompting ( Madaan et al. 2023 ; Kojima et al. 2022 ; Wei et al. 2022 ) , and adding additional reasoning process in GRPO ( Fan et al. 2025 ) , we propose a multi-stage deliberative reasoning framework that encourages more cautious decision-making while creating new supervisable intermediate signals. As illustrated in Fig. 1 , our method guides models through a structured process: first identifying and localizing regions of interest, then performing focused analysis within these regions before producing final decisions. This approach aligns with human expert cognitive strategies during visual inspection, i.e., rapid global scanning followed by focused, detailed examination of suspicious areas. With models now producing detailed spatial predictions, we introduce complementary reward mechanisms that leverage this enriched information. These rewards deliver dual benefits: enhanced reward variance for improved GRPO learning and systematic filtering of spurious correctness patterns. Inspired by multi-objective GRPO frameworks ( Li et al. 2025a ) , we design fine-grained reward functions that assess both decision accuracy and localization quality. Specifically, we prompt the model to not only provide a decision and rationale but also localize the anomalies via bounding boxes. Since defects may spread out on the object of interest, our first reward measures count accuracy of predicted boxes against ground truth, while the second computes Generalized Intersection over Union (GIoU) between predicted and ground-truth boxes. These spatially grounded rewards enable fine-grained discrimination between genuine understanding and spurious correctness, allowing models to exploit available data more effectively while developing reliable visual analysis capabilities. The synergy between multi-stage reasoning and fine-grained rewards creates a comprehensive framework that simultaneously addresses GRPO’s variance dependency and the challenge of supervision for reasoning process, transforming both the learning efficiency and reasoning quality of GRPO-based anomaly detection systems. In summary, we propose a framework for A nomaly D etection via F ine-grained reward optimization and M ulti-stage reasoning ( AD-FM ). Our key contributions are as follows: • We introduce a multi-stage deliberative reasoning framework that creates novel supervisory signals for reasoning processes, enabling systematic visual inspection for interpretable anomaly detection. • We propose a fine-grained reward mechanism that suppresses spurious correctness and maximizes training data utilization through dual-objective optimization of classification accuracy and spatial understanding. • We demonstrate substantial performance improvements across multiple datasets in both in-distribution and out-of-distribution settings, effectively bridging the gap between general-purpose MLLMs and specialized visual inspection requirements. 2 Related Works Anomaly Detection. Anomaly Detection (AD) involves identifying instances that deviate from normal patterns. Modern AD techniques are predominantly unsupervised and aim to model the distribution of normal data ( Zhang et al. 2024b ) . Key approaches include: one-class classification ( Liu et al. 2023 ) , reconstruction-based methods that identify anomalies via reconstruction error ( He et al. 2024 ; Zhang, Xu, and Zhou 2024 ) , feature distillation which compares original and distilled features ( Deng and Li 2022 ) , and embedding-based strategies that measure feature divergence between normal and target samples ( Roth et al. 2022 ; Lei et al. 2023 ) . Zero-shot Anomaly Detection. Zero-shot AD leverages textual prompts to detect anomalies

Read the complete text