Mask Consistency Regularization in Object Removal

Hua Yuan, Jin Yuan, Yicheng Jiang, Yao Zhang, Xin Geng, Yong Rui · 2025

Read Mask Consistency Regularization in Object Removal by Hua Yuan, Jin Yuan, Yicheng Jiang, Yao Zhang, Xin Geng, Yong Rui (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-10259.txt.

Introduction

Removing an object from a photo sounds simple, but Hua Yuan and five co-authors show how image-filling models go wrong in this 2025 computer vision paper. Models often invent unwanted content inside the blanked region, called mask hallucination, or they fill the hole with something shaped like the hole itself, a mask-shape bias.

Their fix, Mask Consistency Regularization, works during training. The mask is perturbed in two ways, stretched larger and reshaped, and the model must keep its filled output consistent with the original mask's result. The stretched masks pull the fill toward the surrounding scene, while reshaped masks break the habit of copying the hole's outline. Experiments show fewer invented objects and cleaner, more believable removals.

Catalog record

CREATORHua Yuan, Jin Yuan, Yicheng Jiang, Yao Zhang, Xin Geng, Yong Rui
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-10259

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Mask Consistency Regularization in Object Removal Hua Yuan, Jin Yuan, Yicheng Jiang, Yao Zhang, Xin Geng, Yong Rui Abstract Object removal, a challenging task within image inpainting, involves seamlessly filling the removed region with content that matches the surrounding context. Despite advancements in diffusion models, current methods still face two critical challenges: (1) mask hallucination, where the model generates irrelevant or spurious content inside the masked region, and (2) mask-shape bias, where the model fills the masked area with an object that mimics the mask’s shape rather than surrounding content. To address these issues, we propose Mask Consistency Regularization (MCR), a novel training strategy designed specifically for object removal tasks. During training, our approach introduces two mask perturbations: dilation and reshape, enforcing consistency between the outputs of these perturbed branches and the original mask. The dilated masks help align the model’s output with the surrounding content, while reshaped masks encourage the model to break the mask-shape bias. This combination of strategies enables MCR to produce more robust and contextually coherent inpainting results. Our experiments demonstrate that MCR significantly reduces hallucinations and mask-shape bias, leading to improved performance in object removal. † † address: 1 School of Computer Science and Engineering, Southeast University, Nanjing, China 2 AI Lab, Lenovo Research, Beijing, China 3 Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China {yuanhua, ycjiang, xgeng}@seu.edu.cn, {jinyuan, yaozhang}@lenovo.com, [email protected] 1 Introduction Image inpainting, particularly for object removal, aims to reconstruct missing regions of an image with visually realistic and semantically coherent content [ 1 , 2 ] . Early deep learning methods adopted convolutional encoder–decoder frameworks and generative adversarial networks (GANs) that leveraged attention mechanisms or partial convolutions to borrow textures from the surrounding context [ 3 , 4 ] . Recent diffusion models, such as Stable Diffusion, have brought significant improvements in visual fidelity and diversity for inpainting [ 5 , 6 ] . Consequently, recent research has predominantly focused on diffusion-based approaches in pursuit of superior image restoration quality [ 7 , 8 ] . In the context of object removal, existing approaches can be broadly categorized into prompt-based and mask-based methods. Prompt-based methods leverage text guidance to specify undesired objects, offering greater flexibility and usability in interactive editing scenarios. For example, Imagen Editor [ 9 ] and InstructPix2Pix [ 10 ] demonstrate that diffusion models can be steered by natural language instructions to remove or modify objects in a controllable manner. On the other hand, mask-based methods exploit explicit spatial priors, often in conjunction with segmentation models, to ensure more precise and reliable object removal. Notably, approaches such as LaMa [ 11 ] and SmartBrush [ 12 ] integrate corrupted image regions and binary masks into the generative process, yielding structurally coherent and semantically faithful completions. These two lines of research highlight the trade-off between flexibility and accuracy in diffusion-based object removal. However, despite these advances, two key challenges remain. First, mask hallucination occurs when the model generates irrelevant or spurious objects within the masked region, introducing unnecessary content beyond the intended completion. Second, mask-shape bias arises when the model interprets the geometric outline of the mask as a semantic cue, producing content that matches the silhouette rather than the surrounding scene. Even worse, the model may regenerate the removed object according to the mask shape, as illustrated in Fig. 2 in Section 3 . These issues highlight the key obstacles to effective object removal. Inspired by the consistency regularization in the semi-supervised learning [ 13 ] , we propose a simple yet effective training framework called Mask Consistency Regularization for diffusion-based object removal. The central idea is to enforce prediction consistency under mask perturbations. Specifically, during training, we introduce two perturbation strategies: (1) mask dilation , which applies morphological dilation [ 14 ] to the original mask, enlarging its region to encourage alignment with the surrounding content, and (2) mask reshape , which constructs random masks or rectangles that cover the original mask area, thereby reducing the model’s dependency on the geometry of the mask. By enforcing consistency between outputs under perturbed and original masks, MCR enhances coherence around mask boundaries while regularizing the inpainting process against mask-specific artifacts. Figure 1: (1) The left part is the overall pipeline of our approach. Given an input image and its corresponding binary mask (outlined in red), we generate two variants of the mask through perturbation: mask dilation and mask reshaping. These perturbed masks, along with the original image, are processed by ControlNet to produce conditional latent encodings ( z_{0},z_{1},z_{2} ). A consistency regularization is enforced between the outputs generated from the original and perturbed masks, guiding the diffusion model to produce coherent and contextually consistent inpainting results. (2) The right part is the illustration of hallucination, mask-shape bias and an ideal output. The proposed method builds upon the Stable Diffusion XL model (SDXL) [ 15 ] and employs a plug-and-play ControlNet module [ 16 ] to guide object removal, where the parameters of SDXL remain entirely frozen to substantially reduce memory usage and training time. Fig. 1 illustrates the training framework of our proposed Mask Consistency Regularization, where perturbed masks (via dilation and reshaping) are applied to the input image. Each of these masked images is processed through ControlNet to produce conditional representations, and consistency regularization is enforced across their outputs, encouraging robust and geometry-agnostic inpainting. Our contributions can be summarized as follows. • We identify two critical challenges in diffusion-based object removal, namely mask hallucination and mask-shape bias and propose an effective framework called Mask Consistency Regularization, which enforces prediction consistency under mask perturbations to alleviate these problems. • We introduce two perturbation strategies: (1) mask dilation , which applies morphological dilation to expand the original region and improve contextual alignment, and (2) mask reshape , which constructs reshaped masks covering the original mask to reduce shape-dependent fillings. • Extensive experiments demonstrate that MCR significantly reduces hallucinations and mask-shape bias compared with existing baselines, and further validate the effectiveness of the proposed two perturbation strategies. 2 Method Our approach frames diffusion-based object removal as a consistency-driven learning problem. Instead of relying solely on reconstruction objectives, we systematically introduce perturbations to the input masks and require the model to produce stable and coherent predictions across these variations, as illustrated in Fig. 1 . This design explicitly decouples the generation process from spurious dependencies on mask geometry, guiding the model to learn semantically meaningful completions rather than memorizing mask boundaries. By incorporating Mask Consistency Regularization into the inpainting framework, our method improves generalization, mitigates mask-induced artifacts, and enhances the overall quality of object removal. 2.1 Preliminary We begin by clarifying the notations used in the task. Let x_{0} denote the original image, M denote the binary mask, and \epsilon represent Gaussian noise added during the forward diffusion process. At a given timestep t , the noisy sample is x_{t} , obtained by perturbing x_{0} with \epsilon . A plug-and-play ControlNet module is employed to guide mask-based object removal. Specifically, the original image x_{0} and the mask M are input to ControlNet, which produces a latent encoding z(x_{0},M) . This encoding, along with the text prompt p , is then used as conditioning

Read the complete text