IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layout

Fei Shen, Xiaoyu Du, Yutong Gao, Jian Yu, Yushe Cao, Xing Lei, Jinhui Tang · 2025

Read IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layout by Fei Shen, Xiaoyu Du, Yutong Gao, Jian Yu, Yushe Cao, Xing Lei, Jinhui Tang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-01949.txt.

Introduction

Fei Shen and co-authors tackle a weak spot in image editing tools in a 2025 computer science paper. Editors based on diffusion models can change what objects mean, but they often move objects, drop some, or add new ones. The team sets a stricter task: change the sense of objects while keeping their exact count and layout. Their system, IMAGHarmony, uses few added parameters and reads cues from the source image.

A harmony aware module feeds those cues into the editing process so the model weighs sense, count, and place together. A second method picks better starting noise, which steadies hard scenes with many objects. The team also built HarmonyBench, a test set for scoring both edit accuracy and layout keeping. In their tests the system beat prior methods on both scores while staying light to run.

Catalog record

CREATORFei Shen, Xiaoyu Du, Yutong Gao, Jian Yu, Yushe Cao, Xing Lei, Jinhui Tang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-01949

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

IMAGHarmony: Controllable Image Editing with Consistent Object Quantity and Layout Fei Shen, Xiaoyu Du, Yutong Gao, Jian Yu, Yushe Cao, Xing Lei, Jinhui Tang Abstract Recent diffusion models have advanced image editing by enhancing visual quality and control, supporting broad applications across creative and personalized domains. However, current image editing largely overlooks multi-object scenarios, where precise control over object categories, counts, and spatial layouts remains a significant challenge. To address this, we introduce a new task, quantity-and-layout consistent image editing (QL-Edit), which aims to enable fine-grained control of object quantity and spatial structure in complex scenes. We further propose IMAGHarmony, a structure-aware framework that incorporates harmony-aware attention (HA) to integrate multimodal semantics, explicitly modeling object counts and layouts to enhance editing accuracy and structural consistency. In addition, we observe that diffusion models are susceptible to initial noise and exhibit strong preferences for specific noise patterns. Motivated by this, we present a preference-guided noise selection (PNS) strategy that chooses semantically aligned initial noise samples based on vision-language matching, thereby improving generation stability and layout consistency in multi-object editing. To support evaluation, we construct HarmonyBench, a comprehensive benchmark covering diverse quantity and layout control scenarios. Extensive experiments demonstrate that IMAGHarmony consistently outperforms state-of-the-art methods in structural alignment and semantic accuracy. The code and model are available at https://github.com/muzishen/IMAGHarmony . 1 Introduction Recent advances in image generation [ 1 , 20 , 35 , 30 , 22 , 27 ] are largely driven by diffusion models [ 7 , 48 , 50 , 67 ] , which outperform traditional methods such as GANs [ 39 , 37 ] and VAEs [ 53 , 8 ] across diverse applications. Building on this progress, prompt-driven image editing [ 41 , 42 , 32 ] allows users to flexibly modify existing images while preserving visual fidelity and semantic alignment. These advances have enabled broad adoption in creative, commercial, and personalized content creation. Most prompt-driven image editing methods [ 68 , 3 , 32 , 54 , 29 , 31 ] fall into two main categories: mask-free [ 68 , 3 , 57 , 17 , 20 ] and mask-based approaches [ 2 , 69 , 65 , 71 , 49 ] . Mask-free approaches [ 68 , 3 , 57 ] rely only on textual prompts to steer the diffusion model, producing global changes without explicit spatial constraints. Mask-based approaches [ 2 , 69 , 65 ] , in turn, confine the edit to user-defined masks, enabling fine-grained updates space between text and reference image. As shown in Figure 1 (a), both paradigms perform well when editing a few-object ( e.g., one or two). However, they [ 41 , 29 , 42 , 32 ] often fail in scenes requiring consistent control over more than two objects, particularly in terms of object counts and spatial layouts from Figure 1 (b). In addition, multi-object editing frequently results in incorrect styles ( e.g., Cyberpunk Style ) and disorganized backgrounds ( e.g., Flowers in Background ). Two primary factors contribute to this limitation: (i) The weak counting capability of diffusion models, where text prompts often fail to convey or retain the desired object quantity; (ii) The inability to distinguish multiple entities from object parts within masked regions, hindering accurate structural and semantic modeling. Figure 1 : Editing results on (a) few-object and (b) multi-object scenes. Existing methods struggle to preserve the count, layout, and semantics in multi-object cases, while ours ensures consistent and faithful edits. Furthermore, recent studies [ 14 , 26 , 35 , 63 , 38 ] have shown that diffusion models are highly susceptible to the choice of initial noise in generation and editing tasks. Since the denoising trajectory [ 56 , 66 , 55 ] is conditioned on the starting noise, different seeds induce distinct attention pathways and layout biases during early stages, leading to substantial variations in object count, spatial arrangement, and visual semantics. For example, InitNO [ 14 ] identifies suboptimal noise as a significant cause of prompt-image misalignment and proposes an attention-based strategy to guide noise toward semantically aligned outputs. Building on these insights [ 64 , 63 , 36 ] , we observe that initial noise is equally critical in multi-object editing, particularly in determining object count and spatial arrangement. Even under identical prompts, different seeds can produce structurally divergent outputs: some yield clean and well-distributed layouts, while others result in missing, overlapping, or misaligned objects. Therefore, identifying effective noise seeds during inference is crucial for ensuring reliable layout and object consistency in multi-object editing. To address the above challenges, we present a new task: quantity-and-layout consistent image editing (QL-Edit). Given a reference image and a textual instruction, the objective is to edit the image such that the output not only aligns with the desired semantics but also strictly conforms to the specified object count and spatial arrangement. To this end, we propose IMAGHarmony, a consistency-aware framework that integrates two key components: a harmony-aware attention (HA) module and a preference-guided noise selection (PNS) strategy. Specifically, the HA module addresses the limitations of existing models in quantity and layout modeling by explicitly encoding object count from auxiliary textual features and implicitly capturing spatial relations from visual features. These representations are fused and injected into the denoising network to guide layout-consistent editing. To mitigate the sensitivity of diffusion models to initial noise, the PNS strategy generates multiple candidate seeds during training. At inference, it performs a few-steps denoising for each seed, evaluates semantic alignment using a frozen vision-language model, and selects the best-performing seed to initialize the complete denoising process, thereby improving stability and structural fidelity. To support evaluation, we further construct HarmonyBench, a comprehensive benchmark tailored for QL-Edit. It includes a wide range of multi-object editing scenarios with diverse object counts, spatial layouts, and editing instructions, enabling systematic assessment of both semantic alignment and structural controllability. Our main contributions are summarized as follows: \bullet We introduce QL-Edit, a new task that enables control image editing while ensuring consistency in object count, category, and spatial layout. \bullet We propose the IMAGHarmony framework, which integrates a HA module and a PNS strategy to enhance structural consistency and generation stability. \bullet We construct HarmonyBench, a benchmark covering diverse quantities, categories, and layouts, along with metrics for structural and semantic evaluation. \bullet Extensive experiments show that IMAGHarmony achieves superior performance over state-of-the-art methods in object count accuracy, layout fidelity, and overall generation quality. 2 Related Work Controllable Image Editing with Diffusion Models. Diffusion models [ 7 , 43 , 48 , 50 , 51 , 45 , 13 ] have shown strong potential for high-fidelity and semantically aligned image generation and editing. Prompt-driven editing methods can be broadly categorized into two paradigms: (1) mask-free editing [ 68 , 3 , 17 ] and (2) mask-based editing. Mask-free approaches [ 3 , 17 , 20 ] rely solely on natural language instructions to guide image transformation without explicit spatial constraints. For instance, MagicBrush [ 68 ] introduces a large-scale manually annotated dataset for real-world instruction-based editing. InstructPix2Pix [ 3 ] trains a conditional diffusion model on synthetic image-instruction pairs generated via GPT-3 [ 10 ] and Stable Diffusion, enabling efficient instruction-following. Prompt-to-Prompt [ 17 ] and SmartEdit

Read the complete text