Multi-Layer Diffusion Strategy for Multi-IP Interaction-Aware Human Erasing
Jinghan Yu, Junhao Xiao, Zhiyuan Ma, Yue Ma, Kaiqi Liu, Yuhan Wang, Daizong Liu, Xianghao Meng, Jianjun Li · 2025
Read Multi-Layer Diffusion Strategy for Multi-IP Interaction-Aware Human Erasing by Jinghan Yu, Junhao Xiao, Zhiyuan Ma, Yue Ma, Kaiqi Liu, Yuhan Wang, Daizong Liu, Xianghao Meng, Jianjun Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-06543.txt.
Introduction
Jinghan Yu and co-authors erase people from photos where crowds overlap and tangle. Their 2025 paper, whose head title names the method MILD, targets the hard cases: people blocking people, hands gripping objects, figures melting into busy backgrounds. Mask-guided erasers fail there because no large training set showed such scenes and the layers are hard to pull apart.
The team built a data set rich in poses, overlaps, and multi-person contact. A new measure counts how much meaning leaks between regions during generation. Their method then splits the work into separate paths, one per person plus one for the background, so each is rebuilt cleanly. Extra guidance from body pose and parsing maps, plus mask-aware attention, sharpens the result.
Catalog record
| CREATOR | Jinghan Yu, Junhao Xiao, Zhiyuan Ma, Yue Ma, Kaiqi Liu, Yuhan Wang, Daizong Liu, Xianghao Meng, Jianjun Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-06543
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-Layer Diffusion Strategy for Multi-IP Interaction-Aware Human Erasing Jinghan Yu, Junhao Xiao, Zhiyuan Ma, Yue Ma, Kaiqi Liu, Yuhan Wang, Daizong Liu, Xianghao Meng, Jianjun Li Abstract Recent years have witnessed the success of diffusion models in image customization tasks. However, existing mask-guided human erasing methods still struggle in complex scenarios such as human–human occlusion , human–object entanglement , and human–background interference , mainly due to the lack of large-scale multi-instance datasets and effective spatial decoupling to separate foreground from background. To bridge these gaps, we curate the MILD dataset capturing diverse poses, occlusions, and complex multi-instance interactions. We then define the Cross-Domain Attention Gap (CAG) , an attention-gap metric to quantify semantic leakage. On top of these, we propose Multi-Layer Diffusion (MILD) , which decomposes the generation process into independent denoising pathways, enabling separate reconstruction of each foreground instance and the background. To enhance human-centric understanding, we introduce Human Morphology Guidance , a plug-and-play module that incorporates pose, parsing, and spatial relationships into the diffusion process to improve structural awareness and restoration quality. Additionally, we present Spatially-Modulated Attention , an adaptive mechanism that leverages spatial mask priors to modulate attention across semantic regions, further widening the CAG to effectively minimize boundary artifacts and mitigate semantic leakage. Experiments show that MILD significantly outperforms existing methods. Datasets and code are publicly available at: project page . Figure 1: The figure exhibits MILD ( our method ) can robustly handle three critical challenges in human erasing: Human–Human Occlusion , Human–Object Entanglement , and Human–Background interferences and can achieve clean and artifact-free removal results across diverse scenarios. 1 Introduction Object removal aims to eliminate user-specified regions and synthesize visually coherent, context-consistent backgrounds ( Jampani et al., 2021 ; Xu et al., 2023 ) . This capability supports a wide range of applications, including photo editing, privacy preservation, and interactive content creation ( Yu et al., 2018 ; Liu et al., 2018 ) . However, achieving both thorough removal and faithful background restoration remains a challenging task, especially in scenarios involving multiple interacting instances and dense occlusions. Early non-parametric methods ( Criminisi et al., 2004 ; Barnes et al., 2009 ) repainted erased regions by copying patches from visible areas, but often produce repetitive textures and fail in complex scenes. Afterwards, GAN-based models ( Pathak et al., 2016 ; Nazeri et al., 2019 ) introduce adversarial learning to enhance visual realism, yet suffer from global inconsistency and instability under large masks. Recently, diffusion-based models ( Song et al., 2021 ; Avrahami et al., 2023 ; Yang et al., 2023a ) have achieved competitive performance by leveraging large-scale pretrained generative priors ( Rombach et al., 2022b ; Ekin et al., 2024 ) . These models ( Zhuang et al., 2024 ; Yildirim et al., 2023 ) excel in generating diverse content, but remain limited in eliminating objects in complex human-centric multi-instance scenarios. Figure 2: Peak signal of semantic leakage. For example, in Figure 2 , standard LDM always fails to remove the target cleanly and instead hallucinates a new human-like figure at the masked region. Notably, the generated content retains dominant visual hints, such as “human”, implying that semantic features from the removed subject have leaked into the generation process. These limitations reveal a fundamental challenge: most existing approaches treat object removal as a unified generation task without explicitly disentangling the generation paths of different foreground instances . This framework often leads to semantic interference and content entanglement, particularly in multi-interactive-person (Multi-IP) or object-dense scenes, where features from one instance may permeate into others, degrading output quality and controllability. To address this challenge, we propose MILD (Multi-Layer Diffusion) strategy, a theoretically grounded framework for human-centric object removal. We first theoretically analyze the source of semantic leakage by introducing the Cross-Domain Attention Gap (CAG) to quantify attention flow discrepancies between foreground and background regions. Theoretical analysis demonstrates that enlarging CAG significantly reduces semantic leakage, inspiring our core design principle: maximizing the attention gap through architectural separation . Guided by this theory, MILD reformulates human erasing as a layered diffusion process, establishing independent diffusion paths for each target instance and the background to achieve instance-level disentanglement. Specifically, MILD employs a shared UNet backbone with domain-specific LoRA adapters, ensuring scalability while maintaining attention domain separation between foreground and background. To enhance instance awareness in complex scenarios, we introduce Human Morphology Guidance (HMG), which injects human pose and parsing priors to preserve structural integrity under occlusion. Furthermore, the Spatially-Modulated Attention (SMA) module applies mask-conditioned constraints to attention logits, further expanding CAG and achieving exponential leakage suppression. This design not only enables more thorough instance removal but also produces flexibly recomposable layered outputs, supporting diverse scene editing operations. To advance research in this field, we release a high-quality human-erasing dataset with diverse scenes, challenging poses, occlusions, and interactions. It provides paired images with precise masks and realistic backgrounds for rigorous real-world evaluation of human-removal models. Experimental results demonstrate that our MILD significantly outperforms existing methods on complex human removal tasks with superior visual realism, semantic consistency, and perceptual quality, confirming its state-of-the-art performance. 2 Related Work We introduce the recent works concentrated in image inpainting (see Appendix C ), object Removal (see Sec. 2 ) and Layered Generation (see Appendix C ). Object Removal Object removal, as a task derived from image inpainting, aims to eliminate user-specified content while preserving scene fidelity. Early approaches ( Suvorov et al., 2022 ) achieved reasonable results by adapting general-purpose generative models with task-specific ( Jain et al., 2023 ; Yi et al., 2020b ) guidance, but they struggled to capture complex textures and scene structures. Recent works ( Hu et al., 2022 ; Kingma and Welling, 2014 ; Ronneberger et al., 2015 ; Rombach et al., 2022a ; Ho et al., 2020 ) have explored a range of strategies to improve controllability and fidelity ( Winter et al., 2024 ) . Training-based ( Yang et al., 2023b ; Jiang et al., 2025 ; Liu et al., 2025 ; Li et al., 2025 ) approaches fine-tune diffusion models using supervision such as masks, prompts, or parameter-efficient modules. Training-free ( Sun et al., 2025 ; Han et al., 2024 ; Jia et al., 2024 ; Li et al., 2024a ) approaches, in contrast, manipulate internal representations (e.g., attention maps or embeddings) during inference to enable flexible editing. In parallel, different forms of user interaction have been introduced—ranging from text-prompted editing ( Bar-Tal et al., 2022 ; Zhao et al., 2021 ; Geng et al., 2024 ) to mask-guided ( Li et al., 2022 ; Köhler et al., 2014 ; Xie et al., 2023a ; Xie et al., 2023b ) generation—to support more intuitive control. Despite these advances, we found that most existing methods lack instance-level distinction, generating a single prediction conditioned on all masked targets or prompts. This entangled treatment limits the compositional flexibility and often leads to interference between each instance and its surrounding region, especially when it comes to complex human-oriented interaction scenes. Figure 3: Overview of MILD. Given an input image and a set of target masks, the proposed Multi-Layer Diffusion (MILD) strategy performs human erasing by generating disentangled foreground layers and a background layer. A shared UNet with Layered LoRA enables