From Plans to Pixels: Learning to Plan and Orchestrate for Open-Ended Image Editing

Anirudh Sundara Rajan, Krishna Kumar Singh, Yong Jae Lee · 2026

Read From Plans to Pixels: Learning to Plan and Orchestrate for Open-Ended Image Editing by Anirudh Sundara Rajan, Krishna Kumar Singh, Yong Jae Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15181.txt.

Introduction

Anirudh Sundara Rajan, Krishna Kumar Singh, and Yong Jae Lee build an experiential framework for open-ended image editing from abstract instructions like making an advertisement more vegetarian-friendly. A planner splits the request into atomic steps. An orchestrator picks tools and regions for each step. A vision-language judge scores instruction adherence and visual quality, training the orchestrator, while winning trajectories refine the planner. Coupling planning to reward-driven execution beats single-step and rule-based multi-step baselines.

Catalog record

CREATORAnirudh Sundara Rajan, Krishna Kumar Singh, Yong Jae Lee
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90531
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15181

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Plans to Pixels: Learning to Plan and Orchestrate for Open-Ended Image Editing Anirudh Sundara Rajan, Krishna Kumar Singh, Yong Jae Lee Abstract Modern image editing models produce realistic results but struggle with abstract, multi-step instructions (e.g., “make this advertisement more vegetarian-friendly”). Prior agent-based methods decompose such tasks but rely on handcrafted pipelines or teacher-imitation, limiting flexibility and decoupling learning from actual editing outcomes. We propose an experiential framework for abstract, long-horizon image editing , where a planner generates structured atomic decompositions and an orchestrator selects tools and regions to execute each step. A vision–language judge provides outcome-based rewards for instruction adherence and visual quality. The orchestrator is trained to maximize these rewards, and successful trajectories are used to refine the planner. By tightly coupling planning with reward-driven execution, our approach yields more coherent and reliable edits than single-step or rule-based multi-step baselines. Project Page: https://anisundar18.github.io/Plan2Pix.github.io/ 1 Introduction Recent advances in diffusion-based image editing have significantly improved the fidelity and controllability of instruction-based visual modifications. Methods such as InstructPix2Pix [ 2 ] , Prompt-to-Prompt [ 10 ] , and large-scale editors like Flux Kontext [ 19 ] and Qwen-Image-Edit [ 47 ] perform well on well-specified edits (e.g., “add a hat to the man” , “change the car color to red” ), where the instruction corresponds to a simple concrete transformation. Figure 1 : Given a high-level instruction such as “Adapt for a rural audience,” single-step editors (e.g., Flux Kontext [ 19 ] and Qwen-Image-Edit [ 47 ] ) struggle to jointly adapt visual themes, textual content, and audience-specific context while preserving the original advertisement layout and identity. In contrast, our framework decomposes the task into structured subtasks and orchestrates multiple tools using outcome-based feedback. The four images to the right of our result illustrate intermediate edits produced by automatically selected tool–region combinations during orchestration (corresponding subtasks are: (1) Replace the background with a rural scene featuring a farmer holding a basket of produce, while keeping the existing text elements intact ; (2) Replace the text ‘FLY TOWARDS THE BEST FUTURE’ with ‘STRIVE FOR A BETTER TOMORROW’ ; (3) Replace ‘Cent Vidhyarthi’ with ‘Rural Education Loan’ ; and (4) Add a village house or temple in the background to reinforce the rural setting ). However, many real-world editing tasks are abstract, open-ended, and long-horizon. For example, adapting a student-focused loan advertisement into a campaign targeting rural audiences (Fig. 1 ) requires coordinated changes to imagery, slogans, audience-specific messaging, and environmental context—far beyond a single atomic edit. Different subtasks may also require different tools (e.g., object replacement vs. text modification). Prior agent-based systems attempt multi-step orchestration but often rely on handcrafted pipelines or teacher-imitation [ 52 , 62 , 17 , 53 ] , fixing execution order and heuristics. These approaches do not train the planner on its own distribution and do not optimize tool selection based on actual editing outcomes, which can lead to distribution shift, limited generalization, and poor scalability to open-ended instructions. To address these limitations, we decouple long-horizon image editing into planning and orchestration . Given a high-level abstract instruction, the planner produces a checklist-guided decomposition into atomic subtasks and is trained on its own sampled plans to reduce distribution shift and improving stability relative to teacher imitation. Conditioned on the plan, the orchestrator selects tools and regions, executes edits, and receives outcome-based feedback from a VLM judge evaluating instruction adherence, identity preservation, and visual quality. These rewards directly supervise tool selection, grounding decisions in empirical performance. A refinement stage prunes infeasible subtasks, aligning plans with executable actions. Together, this forms an experiential learning framework that improves through interaction with editing tools and judged outcomes. Training this system, however, poses challenges beyond standard supervision: there is no large-scale dataset of abstract multi-step plans, tool selection is context-dependent and ambiguous, and multiple edited outputs can validly satisfy the same instruction. In addition, invoking modern image editing tools is computationally expensive making exploration intractable. These factors make fixed-label standard supervised training challenging. We therefore adopt an experiential learning paradigm grounded in observed editing outcomes. To keep training tractable, we approximate trajectory reward as the sum of independently evaluated sub-task rewards, enabling precomputation over tool–region pairs. The planner learns structured decompositions via checklist-guided self-supervision, while the orchestrator learns tool and region selection directly from judged edits rather than prompts or teacher traces. This design removes handcrafted rules, aligns training with inference, and improves generalization to open-ended instructions. Extensive experiments demonstrate that our framework produces more reliable, coherent, and instruction-faithful results than both single-step generation approaches and multi-step agent baselines. Our key contributions are: • Long-horizon, high-level image editing framework. We cast abstract, open-ended editing as a coordinated planning-and-orchestration problem, enabling multi-step reasoning beyond single-step generation. • Self-Supervised checklist-guided plan generation. A structured planner learns multi-step decompositions from its own checklist-guided samples, reducing distribution shift. • Experiential orchestrator. A reward-driven policy jointly selects tools and regions based on judged executed edits, grounding decisions in empirical outcomes rather than handcrafted rules. • Closed-loop refinement and strong results. We prune infeasible sub-tasks using orchestration feedback and achieve state-of-the-art performance for open-ended image editing. 2 Related Work Controllable Image Editing with Diffusion Models. Diffusion-based models have achieved strong performance in text-guided image editing [ 37 , 35 ] . Training-free methods such as SDEdit and Prompt-to-Prompt [ 28 , 10 , 34 , 3 , 11 ] manipulate the denoising process for prompt-aligned edits, but are typically limited to localized changes and may over-edit or under-follow instructions. Training-based approaches, including InstructPix2Pix and MagicBrush [ 2 , 58 ] , improve robustness via paired supervision. Later methods add control signals (e.g., masks, boxes, drag-based inputs) to enhance spatial precision [ 20 , 43 , 29 , 40 , 30 ] . However, these systems assume well-specified, low-level instructions and often require manual controls. In contrast, we target abstract, open-ended instructions requiring multi-step reasoning and coordinated tool use. Multimodal LLMs for Image Editing and Planning. In vision, recent work generates code to invoke specialized modules, decomposing tasks into tool-executable subproblems [ 9 , 41 , 13 , 15 ] . These systems treat pretrained models as callable tools and use LLMs to orchestrate their composition for complex visual reasoning. Building on this paradigm of task decomposition and tool invocation, multimodal LLMs (MLLMs) extend language models with visual inputs for joint text–image reasoning [ 23 , 63 , 22 ] , and have recently been applied to image editing. For example, MGIE [ 5 ] rewrites instructions before passing them to a diffusion editor, while other systems use VLM agents to decompose complex editing requests into simpler steps executed by a fixed editor [ 52 , 62 , 17 , 53 ] . These approaches are typically training-free or rely on imitation of teacher plans, and do not learn from the outcomes of real edits—planners are not trained on their own plan distributions, and tool selection is not policy-optimized. In contrast, our framework couples checklist-guided planning with experiential orchestration, learning tool and region selection directly from judged editing outcomes. Experiential Learning for Long-Horizon Reasoning. Reinforcement learning (RL) has recently been used to enhance long-horizon reasoning in language models, enabling step decomposition, iterative refinement, and improved robustness [

Read the complete text