Beyond Simple Edits: X-Planner for Complex Instruction-Based Image Editing
Chun-Hsiao Yeh, Yilin Wang, Nanxuan Zhao, Richard Zhang, Yuheng Li, Krishna Kumar Singh · 2025
Read Beyond Simple Edits: X-Planner for Complex Instruction-Based Image Editing by Chun-Hsiao Yeh, Yilin Wang, Nanxuan Zhao, Richard Zhang, Yuheng Li, Krishna Kumar Singh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-05259.txt.
Introduction
Chun-Hsiao Yeh and co-authors present a planner that helps image editing software follow tangled, indirect requests, in a 2025 paper. A user might ask for a photo fix that hides three changes in one sentence. Editing models often miss the point, change faces by accident, or need a hand-drawn mask. The planner, X-Planner, reads the request with a large multimodal model.
It thinks step by step and splits the request into plain sub-tasks. For each one it names the edit type and draws the mask itself, so edits stay local and faces stay true. The team also built an automatic pipeline to make training data at scale. They report top scores on older test sets and on a new, harder one they made for tangled requests.
Catalog record
| CREATOR | Chun-Hsiao Yeh, Yilin Wang, Nanxuan Zhao, Richard Zhang, Yuheng Li, Krishna Kumar Singh |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 62 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-05259
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Beyond Simple Edits: X-Planner for Complex Instruction-Based Image Editing Chun-Hsiao Yeh, Yilin Wang, Nanxuan Zhao, Richard Zhang, Yuheng Li, Krishna Kumar Singh Abstract Recent diffusion-based image editing methods have significantly advanced text-guided tasks but often struggle to interpret complex, indirect instructions. Moreover, current models frequently suffer from poor identity preservation, unintended edits, or rely heavily on manual masks. To address these challenges, we introduce X-Planner , a Multimodal Large Language Model (MLLM)-based planning system that effectively bridges user intent with editing model capabilities. X-Planner employs chain-of-thought reasoning to systematically decompose complex instructions into simpler, clear sub-instructions. For each sub-instruction, X-Planner automatically generates precise edit types and segmentation masks, eliminating manual intervention and ensuring localized, identity-preserving edits. Additionally, we propose a novel automated pipeline for generating large-scale data to train X-Planner which achieves state-of-the-art results on both existing benchmarks and our newly introduced complex editing benchmark. The project page is available at https://danielchyeh.github.io/x-planner/ . Figure 1 : Left. Given a source image and complex instruction, our MLLM based X-Planner decomposes the complex instruction into simpler sub-instructions (with edit type) along with auto-generated segmentation masks indicating the editing regions (shown in bottom left of each edited image) and hallucinates additional bounding box of object for the insertion case. We iteratively perform localized editing, by providing X-Planner ’s editing instruction and region (mask and box) to compatible editing model for each edit type. Right. Recent SmartEdit [ 16 ] and MGIE [ 11 ] which also use MLLM struggles with complex instruction understanding and identity preservation. † † ∗ Work done during CHY’s summer internship at Adobe Research. 1 Introduction The field of generative image editing has experienced remarkable advancements in recent years [ 27 , 24 , 14 , 3 , 5 , 4 ] , driven by the development of diffusion models [ 34 , 31 ] . Broadly, these advancements in image editing can be categorized into two categories. The first involves free-form editing, adapts pre-trained diffusion models to edit images based solely on text instructions and a source image [ 4 , 47 , 37 ] . These methods often suffer from over-editing, modifying regions beyond those intended by the user. Subsequent research involve controllable image editing that incorporates user-provided control signals—such as semantic segmentation masks [ 27 , 3 , 50 ] , bounding boxes [ 6 , 21 , 42 ] , content dragging and blobs [ 26 , 38 , 28 ] , and image prompts [ 46 , 7 ] to guide task-specific edits. These signals improve editing control but are time-consuming for users to provide manually. Another critical challenge of these models is to robustly interpret and execute complex instructions. Existing methods often struggle with nuanced requirements of these complex instructions, which limits their ability to to perform these edits effectively and provide intuitive, user-friendly interactions to allow more direct user controls over these edits. Below, we identify several key challenges that highlight limitations of current editing models: (1) Multi-Object Targeting Instructions: A single instruction might targets multiple objects within an image which requires editing model to identify each object which needs to be edited according to this single instruction and image content (e.g., first row in Figure 1 ). (2)Multi-Task Instructions: Instructions with multiple distinct edits within a single prompt (e.g., second row in Figure 1 ). (3) Indirect Instruction Interpretation: Instructions that contain indirect cues which require deeper understanding and decomposition into multiple steps to achieve accurate results (e.g., last row in Figure 1 ). Recent diffusion-based image editing methods have significantly advanced text-guided tasks, yet still struggle to robustly interpret and execute complex, indirect instructions. Typically, these tasks require extensive manual effort to simplify instructions and provide precise region guidance, limiting scalability and usability. While MLLMs offer promise due to extensive world knowledge, directly applying them often results in misinterpretation and localization errors (as shown in Figure 1 , SmartEdit [ 16 ] and MGIE [ 11 ] struggle to perform these complex edits as it misunderstand the editing prompt ,e.g., SmartEdit generates ice-cream, instead making it summer scene). One solution is to force these MLLMs to reason about these complex instructions based on image content and apply chain-of-thought reasoning to autonomously break down complex instructions into simpler sub-instructions aligned with image context. In addition, we need these MLLMs to provide region guidance in terms of masks or bounding boxes for these edits. Some existing MLLMs like LISA [ 20 ] and GLaMM [ 32 ] provide grounding masks while answering image-related questions but have not been trained for localizing editing instructions. Recent work, GenArtist [ 43 ] , explored this direction by leveraging closed-source GPT-4 [ 1 ] for instruction decomposition. However, it has several key limitations: (1) It focuses on breaking down long, nested prompts rather than reasoning about complex and indirect instructions, limiting its effectiveness for intricate editing tasks. (2) GenArtist relies on closed-source model and external toolboxes even during inference, restricting adaptability for users and hindering fine-tuning or further research advancements for the community. (3) Most importantly, its dependence on external object detectors [ 23 ] and segmentation model [ 19 ] results in bounding boxes and masks that are not optimized for different editing types, leading to failures in tasks requiring more than simple object detection. For instance, in object insertion tasks (e.g., ”add a cat”) where the cat is not originally present in the image, external detectors and segmentation models (e.g., SAM [ 19 ] ) struggle to hallucinate and localize the inserted object, failing to provide effective editing guidance. To address these challenges, we propose X-Planner , a Multimodal Large Language Model (MLLM)-driven planning system that excels in managing complex instruction-based image editing tasks as shown in Figure 1 . Our X-Planner operates by breaking down intricate user instructions into structured, simpler sub-instructions, each accompanied by auto-generated edit type and mask corresponding to main editing anchor (e.g., in second row of Figure 1 , “local texture: Make < tree > to be in cyberpunk ” indicates the edit type to be of texture change and main editing anchor is tree for which we generate mask). This decomposition strategy empowers the model to interpret ambiguous requests and apply stepwise complex edits. Our approach refines spatial control by tailoring masks to each edit type—tight for color/texture changes, coarse for replacements, and full-image for global edits—preventing over-editing and preserving content outside the mask. In the case of insertion edits, just the mask for editing anchor is not sufficient as the inserted object region can be outside the mask. As shown in Figure 1 , the inserted ornaments have to be around the cat which is beyond the mask of cat’s body, for which GenArtist [ 43 ] by relying on external detector and segmentor would often fail to handle. X-Planner addresses this issue by leveraging the MLLM world knowledge and reasoning about the image content predicts an additional bounding box indicating the region where the object could be inserted. X-Planner also predicts edit type which enables dynamically selecting the most suitable editing model for each edit type. Figure 2 : Overview of X-Planner for Complex Instruction-Based Editing. Our X-Planner comprises two main branches: First, MLLM decomposes the