Weakly-supervised Localization of Manipulated Image Regions Using Multi-resolution Learned Features

Ziyong Wang, Charith Abhayaratne · 2025

Read Weakly-supervised Localization of Manipulated Image Regions Using Multi-resolution Learned Features by Ziyong Wang, Charith Abhayaratne (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23586.txt.

Introduction

Ziyong Wang and Charith Abhayaratne locate doctored regions in images without pixel-level labels in a 2025 paper. Editing tools are everywhere, and finding exactly which pixels were faked matters as much as flagging the image. Full supervision needs costly pixel annotations that real cases rarely have. Their method works from image-level verdicts alone.

A detection network first yields activation maps from several learned views, which are fused into a rough suspect map. Pre-trained segmentation models then supply crisp region boundaries, and Bayesian inference merges the two sources into a final mask. Experiments show the pipeline finds manipulated regions credibly despite never seeing a hand-drawn mask. The authors offer it as a practical route for forensic triage of suspect images.

Catalog record

CREATORZiyong Wang, Charith Abhayaratne
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23586

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Weakly-supervised Localization of Manipulated Image Regions Using Multi-resolution Learned Features Ziyong Wang, Charith Abhayaratne Abstract The explosive growth of digital images and the widespread availability of image editing tools have made image manipulation detection an increasingly critical challenge. Current deep learning-based manipulation detection methods excel in achieving high image-level classification accuracy, they often fall short in terms of interpretability and localization of manipulated regions. Additionally, the absence of pixel-wise annotations in real-world scenarios limits the existing fully-supervised manipulation localization techniques. To address these challenges, we propose a novel weakly-supervised approach that integrates activation maps generated by image-level manipulation detection networks with segmentation maps from pre-trained models. Specifically, we build on our previous image-level work named WCBnet to produce multi-view feature maps which are subsequently fused for coarse localization. These coarse maps are then refined using detailed segmented regional information provided by pre-trained segmentation models (such as DeepLab, SegmentAnything and PSPnet), with Bayesian inference employed to enhance the manipulation localization. Experimental results demonstrate the effectiveness of our approach, highlighting the feasibility to localize image manipulations without relying on pixel-level labels. Weakly-supervised Manipulation localization † † email: [email protected] † † email: [email protected] † † affiliation: Department of Electronic and Electrical Engineering, The University of Sheffield, S1 3JD Sheffield, United Kingdom 1 Introduction With the rapid growth and increasing accessibility of image manipulation tools, manipulated images are being produced at an alarming rate, posing a significant threat to cyber security [ Thakur and Rohilla(2020) , Chennamma and Madhushree(2023) ] . Effective detection of manipulated images is vital for maintaining the integrity of visual information across domains such as journalism, legal evidence, and social media [ Zanardelli et al.(2023)Zanardelli, Guerrini, Leonardi, and Adami , Tyagi and Yadav(2023) ] . In recent years, deep learning has been commonly used for image manipulation detection which enables remarkably high performance at either image-level classification [ Camacho and Wang(2022) , Hamid et al.(2023)Hamid, Elyassami, Gulzar, Balasaraswathi, Habuza, and Wani ] or pixel-level localization [ Guo et al.(2023)Guo, Liu, Ren, Grosz, Masi, and Liu , Barni et al.(2021)Barni, Phan, and Tondi ] . However in real world, the manipulated images typically do not have manually annotated pixel-level labels. It prevents the image-level methods from being further leveraged to precisely localize manipulated regions and provide sufficient interpretability [ Walia et al.(2022)Walia, Kumar, Agarwal, and Kim , Abir et al.(2023)Abir, Khanam, Alam, Hadjouni, Elmannai, Bourouis, Dey, and Khan ] . The absence of annotated data also restricts the effectiveness of current fully-supervised manipulation localization methods [ Zeng et al.(2024)Zeng, Wang, Zhou, Zhang, and Meng ] . Thus, the challenge of accurately localizing manipulated regions within images without pixel-level labels remains challenging. Several weakly-supervised image manipulation localization methods have been proposed in recent years. The activation maps of image-wise manipulation model, extracted by grad-CAM, are directly applied as pixel-wise predictions and compared with semi- or fully-supervised models [ Țânțaru et al.(2024)Țânțaru, Oneață, and Oneață ] . Object edges extracted by clustering super-pixels are applied to enhance the grad-CAM result in order to localize the manipulated regions [ Zhou et al.(2024)Zhou, Wang, Zeng, Zhang, and Meng ] , while another method applies a three-source stream (RGB, SRM and Bayar) to generate pseudo pixel-wise labels leveraging the image-wise model for manipulation localization [ Zhai et al.(2023)Zhai, Luan, Doermann, and Yuan ] . However, enhancement through image segmentation is more closely aligned with the nature of image manipulation itself, as it involves the insertion of meaningful objects or regions to create misinformation [ Zhang et al.(2020)Zhang, Zhou, Zhao, Man, Liu, and Yao ] . Segmentation-based approaches [ Hacıefendioğlu et al.(2024)Hacıefendioğlu, Adanur, and Demir , Hasany et al.(2023)Hasany, Petitjean, and Mériaudeau , Kweon et al.(2023)Kweon, Yoon, and Yoon ] can provide pixel-level details but typically require extensive labelled data, which is often unavailable. Consequently, the segmentation models require collaboration with image manipulation detection methods for distinguishing the manipulated class. This paper addresses the challenge of localizing image manipulations without pixel-level annotations by proposing a novel method that combines the multi-view activation map of the classification network with the fine-grained region captures of pre-trained segmentation models. Specifically, our image-wise manipulation network is built on a structure named Cross-block Attention Module (CBAM) from our previous work, an image-wise manipulation method WCBnet [ Wang and Abhayaratne(2023) ] , which weights and fuses the convolutional block output feature at single fixed receptive field. The multi-view activation map is obtained by computing grad-CAMs [ Selvaraju et al.(2020)Selvaraju, Cogswell, Das, Vedantam, Parikh, and Batra ] on differently-fused features from varying configurations of CBAMs, the geometric mean of which involves multi-resolution activation maps across comprehensive receptive fields. This multi-view activation map is considered as coarse localization, which is further leveraged by several pre-trained segmentation models, such as DeepLab [ Weber et al.(2021)Weber, Wang, Qiao, Xie, Collins, Zhu, Yuan, Kim, Yu, Cremers, Leal-Taixe, Yuille, Schroff, Adam, and Chen ] , SAM [ Kirillov et al.(2023)Kirillov, Mintun, Ravi, Mao, Rolland, Gustafson, Xiao, Whitehead, Berg, Lo, Dollár, and Girshick ] , and PSPnet [ Gupta(2023) ] that segment the images into potentially-manipulated regions. Eventually, the activation maps are integrated with these segmentation maps via Bayesian inference, thereby generating more accurate manipulation localization results. The primary contribution of this work is the demonstration of the feasibility for accurate image manipulation localization without requiring pixel-level annotations. This is accomplished by the innovative combination of activation maps from image classification networks and region masks from pre-trained segmentation networks. 2 Methodology Figure 1: The work-flow of proposed weakly-supervised manipulation localization model; Each step is labeled and presented in different colors. In this section, we describe the methodology employed to detect image manipulation and enhance the localization of manipulated regions without using pixel-wise ground-truth labels. As shown in the Figure 1 , our approach consists of four main steps: image-wise manipulation classification, feature map generation, segmentation map extraction using a pre-trained network, and combining these outputs to produce enhanced heatmaps. 2.1 Image-wise Image Manipulation Classification The first step in our method involves classifying whether an image has been manipulated or pristine. In this study, a CNN-based feature extractor is employed to extract the hierarchical features from input images which contain global and local information of manipulation traces. Since both types of information are equally important in image manipulation detection, we apply Cross-block Attention Module (CBAM) from an image manipulation detection network (called WCBnet from our previous work [ Wang and Abhayaratne(2023) ] ) to assign adaptive weights for these high-level and low-level features and fuse them. Specifically, the input three-channel (RGB) image of shape x\in\mathbb{R}^{384\times 384\times 3} is first processed by a CNN-based feature extractor (ResNet50) to produce multi-scale feature maps \{F_{i}\}_{i=1}^{5} , where each F_{i} represents features extracted by 5 convolutional blocks at different scales. F_{i}\in\mathbb{R}^{H_{i}\times W_{i}\times K_{i}} that represents the feature map at scale i is defined as: F_{i}=\text{CNN}(x). (1) These feature maps are then fed into the CBAM structure, which rescales the features to a consistent dimensionality \mathbb{R}^{H\times W\times D} and applies self-attention mechanisms to model the relationships between different convolutional blocks. This process assigns trainable weights according to the inner relationships between features, enhancing the model’s ability to capture and utilize complex feature dependencies. These feature maps are weighted and concatenated as: F_{c}=\text{CBAM}(F_{1},F_{2},\ldots,F_{N}). (2) The processed features

Read the complete text