Structure-Preserving Zero-Shot Image Editing via Stage-Wise Latent Injection in Diffusion Models

Dasol Jeong, Donggoo Kang, Jiwon Park, Hyebean Lee, Joonki Paik · 2025

Read Structure-Preserving Zero-Shot Image Editing via Stage-Wise Latent Injection in Diffusion Models by Dasol Jeong, Donggoo Kang, Jiwon Park, Hyebean Lee, Joonki Paik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15723.txt.

Introduction

Dasol Jeong, Donggoo Kang, Jiwon Park, Hyebean Lee, and Joonki Paik present a 2025 method for editing images with diffusion models while keeping the frame intact. Editors want text-led or reference-led changes with no retraining and no warped layout. The method inverts the source image into the model's code space, then injects codes in stages. Shape goes in early, surface traits late, so edits land where they belong.

Reference codes join through the model's cross-attention to align meaning. Tests on face swaps of expression, texture shifts, and style pours topped past methods. Stage timing, the work shows, is the trick that keeps a photo itself while its look changes. Consumer edit tools chase exactly this balance.

Catalog record

CREATORDasol Jeong, Donggoo Kang, Jiwon Park, Hyebean Lee, Joonki Paik
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-15723

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Structure-Preserving Zero-Shot Image Editing via Stage-Wise Latent Injection in Diffusion Models Dasol Jeong, Donggoo Kang, Jiwon Park, Hyebean Lee, Joonki Paik Abstract We propose a diffusion-based framework for zero-shot image editing that unifies text-guided and reference-guided approaches without requiring fine-tuning. Our method leverages diffusion inversion and timestep-specific null-text embeddings to preserve the structural integrity of the source image. By introducing a stage-wise latent injection strategy—shape injection in early steps and attribute injection in later steps—we enable precise, fine-grained modifications while maintaining global consistency. Cross-attention with reference latents facilitates semantic alignment between the source and reference. Extensive experiments across expression transfer, texture transformation, and style infusion demonstrate state-of-the-art performance, confirming the method’s scalability and adaptability to diverse image editing scenarios. † † This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. 1 Introduction Image editing is a fundamental task in computer vision, with applications in digital art, content creation, and scientific visualization Rombach et al. 2022 ; Meng et al. 2021 ; Hertz et al. 2022 . Traditional editing methods often require manual intervention and domain-specific expertise, limiting accessibility and scalability. Recent advancements in diffusion models have enabled high-quality, automated image editing, but existing approaches still face limitations in structural fidelity and flexible attribute control. Figure 1 : Results of reference-guided image editing using the proposed method. Text-guided image editing, powered by models such as CLIP Radford et al. 2021 , allows users to modify images based on natural language descriptions. While intuitive, these methods Avrahami et al. 2022 ; Nichol et al. 2021 ; Kim et al. 2022 often lack fine-grained control and fail to preserve the structural integrity of the source image, particularly in localized modifications. Conversely, reference-guided editing Ruiz et al. 2023 ; Dong et al. 2022 ; Gal et al. 2022 ; Hu et al. 2021 transfers attributes from an example image but frequently distorts the original structure and requires fine-tuning for diverse tasks. These challenges highlight the need for a unified framework that can effectively combine text-based and reference-based modifications while ensuring structural consistency. A widely used technique in real image editing is DDIM inversion Song et al. 2020a , which reconstructs the latent representation of an image by reversing the denoising process. This allows generative models to manipulate real images in a controlled manner Mokady et al. 2023 ; Miyake et al. 2023 ; Han et al. 2024 ; Zhang et al. 2024 ; Cao et al. 2023 ; Elarabawy et al. 2022 . A promising direction is Null-Text Inversion (NTI) Mokady et al. 2023 , which enables text-guided editing while preserving the structure of real images. However, it is inherently limited to textual modifications and does not support reference-based attribute transfer. Similarly, RIVAL Zhang et al. 2024 improves inversion-based structural alignment but struggles with localized fine-grained attribute control, leading to unwanted distortions when transferring styles or textures from reference images. These challenges necessitate an approach that seamlessly integrates text and reference guidance while preserving structural integrity. To address these limitations, we propose a zero-shot image editing framework that unifies text-guided and reference-guided methods using a stage-wise injection strategy within a diffusion-based architecture. Our approach introduces self-attention on the reference image in the early timesteps to preserve spatial structure, ensuring that the edited output maintains the form and layout of the source image. As the denoising process progresses, attribute injection via self-attention on the reference latents enables fine-grained semantic and stylistic transfer, maintaining coherence with the source image while ensuring that the transferred attributes are contextually aligned rather than arbitrarily imposed. Fig. 1 illustrates the effectiveness of our approach, demonstrating how our method seamlessly integrates desired modifications without the need for masks, fine-tuning, or additional constraints. Unlike prior methods that require explicit region annotations or heavy architectural modifications, our framework accurately transforms only the intended regions while preserving the global structure of the image. The results highlight the robustness of our stage-wise shape and attribute injection, ensuring that fine-grained transformations are both structurally coherent and semantically meaningful. Additionally, our approach optimizes null-text embeddings to act as a structural reference, preventing unwanted deformations during editing. Unlike existing methods that treat null-text embeddings as simple placeholders, we dynamically adjust them based on diffusion timesteps, ensuring that structural consistency is maintained while enabling controlled semantic modifications. By leveraging this combination of structural preservation and fine-grained control, our framework achieves high-quality zero-shot image editing without requiring task-specific fine-tuning. Extensive experiments on AFHQ Choi et al. 2020 and Oxford-IIIT Pets Parkhi et al. 2012 demonstrate state-of-the-art performance in perceptual quality, semantic consistency, and structural preservation. Our method outperforms existing approaches in both zero-shot reference- and text-guided editing tasks, establishing a new benchmark for diffusion-based image editing. 2 Related Work 2.1 Diffusion Models for Image Editing Diffusion models have demonstrated remarkable capabilities in generating high-quality and semantically coherent images Ho et al. 2020 ; Song et al. 2020b ; Rombach et al. 2022 ; Podell et al. 2023 . By operating in a compressed latent space, these models have significantly improved the efficiency and scalability of image synthesis and editing. Text-guided image editing Meng et al. 2021 ; Saharia et al. 2022 has achieved remarkable progress, leveraging models like CLIP for intuitive natural language-based manipulation. Despite these advancements, such methods often lack fine-grained control and struggle to preserve structural integrity during complex edits, limiting their applicability in tasks requiring precise transformations. Reference-guided image editing Gal et al. 2022 ; Ruiz et al. 2023 ; Dong et al. 2022 provides strong attribute transfer capabilities but frequently distorts structural integrity, especially when transferring complex features such as textures or expressions. Many of these methods require extensive fine-tuning, making them less practical for zero-shot scenarios. To enhance control, diffusion-based generative models have increasingly incorporated mask-guided techniques Avrahami et al. 2022 ; Nichol et al. 2021 ; Hertz et al. 2022 for tasks such as inpainting, attribute manipulation, and structure-preserving editing. However, these approaches often rely on user-specified masks for precise control, introducing additional complexity and reducing their practicality in zero-shot settings. 2.2 DDIM Inversion for Real Image Editing A key challenge in real image editing is reconstructing an image in the diffusion latent space while preserving its original structure. DDIM Inversion Song et al. 2020a ; Dhariwal and Nichol 2021 enables this by reversing the forward diffusion process, allowing fine-grained edits while maintaining the global structure. Null-Text Inversion (NTI) Mokady et al. 2023 builds upon DDIM inversion to preserve structure during text-guided editing. By optimizing null-text embeddings, NTI allows controlled semantic modifications without requiring fine-tuning. However, it is inherently limited to text-based prompts and cannot integrate external reference attributes. RIVAL Zhang et al. 2024 extends DDIM inversion to better align structural representations across real images. While effective at handling domain shifts, RIVAL still struggles with fine-grained attribute transfer, particularly in localized modifications such as expression changes or texture blending. Although these methods improve real image editing, they remain either constrained to text guidance (NTI) or prone to distortions in reference-guided tasks (RIVAL). This limitation motivates our stage-wise injection framework, which preserves structural consistency while enabling precise attribute transfer. 2.3 Image-to-Image Translation in Diffusion Models Recent diffusion-based image-to-image

Read the complete text