SMART-Editor: A Multi-Agent Framework for Human-Like Design Editing with Structural Integrity

Ishani Mondal, Meera Bharadwaj, Ayush Roy, Aparna Garimella, Jordan Boyd-Graber · 2025

Read SMART-Editor: A Multi-Agent Framework for Human-Like Design Editing with Structural Integrity by Ishani Mondal, Meera Bharadwaj, Ayush Roy, Aparna Garimella, Jordan Boyd-Graber (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23095.txt.

Introduction

Ishani Mondal and co-authors edit posters, websites, and photos with a team of agents that guards the whole layout while changing parts, in work from twenty twenty-five. Most editing models touch local spots and break global coherence. Human editors plan the full design as they go.

Their framework refines edits at run time under reward guidance, and also trains with preference pairs aligned to those rewards. A new benchmark grades cascading edits across structured and natural images. The system beat strong baselines, with training time tuning gaining most on structured designs. Run time refinement shone on natural photos. Human judges backed the scores, crediting reward guided planning.

Catalog record

CREATORIshani Mondal, Meera Bharadwaj, Ayush Roy, Aparna Garimella, Jordan Boyd-Graber
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-23095

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SMART-Editor: A Multi-Agent Framework for Human-Like Design Editing with Structural Integrity Ishani Mondal, Meera Bharadwaj, Ayush Roy, Aparna Garimella, Jordan Boyd-Graber Abstract We present SMART-Editor , a framework for compositional layout and content editing across structured (posters, websites) and unstructured (natural images) domains. Unlike prior models that perform local edits, SMART-Editor preserves global coherence through two strategies: Reward-Refine, an inference-time reward-guided refinement method, and RewardDPO, a training-time preference optimization approach using reward-aligned layout pairs. To evaluate model performance, we introduce SMARTEdit-Bench , a benchmark covering multi-domain, cascading edit scenarios. SMART-Editor outperforms strong baselines like InstructPix2Pix and HIVE, with RewardDPOachieving up to 15% gains in structured settings and Reward-Refine showing advantages on natural images. Automatic and human evaluations confirm the value of reward-guided planning in producing semantically consistent and visually aligned edits. 1 Introduction Dataset / System Domain(s) Edit-Aware Comp. Reasoning Cascading Instruction Eval Scope Notable Features InstructEdit Wang et al. 2023 Natural Images ✓ – – Natural Language Visual, Mask Accuracy LLM + segmentation + SD for multi-object edits. PostScoreEdit Social Media ✓ – – Prompt Paraphrasing Engagement, Human Ratings Paraphrase+rank to boost audience engagement. FFCLIP Radford et al. 2021 Faces, Cars, Churches ✓ Partial – Free-form Text CLIP Alignment StyleGAN latent editing via CLIP alignment. DocEdit-v2 Suri et al. 2024 Structured Docs ✓ Local – Free/Structured DOM Tree, CSS IoU LMM-based HTML editing + reformulation. LayoutGPT Feng et al. 2023b 2D/3D Layouts – ✓ – Prompt Only Layout Plausibility CSS-like layout synthesis with spatial reasoning. COLE Jia et al. 2024 Graphic Design – Hierarchical – Design Intent Visual + Typographic JSON + image gen for typography and layout. SciPostLayout Tanaka et al. 2024 Scientific Posters – – – Layout Only Visual Structure Poster layout generation only, no edit modeling. LayoutCoT Shi et al. 2025 Posters, UI Layouts – CoT Only – CoT Prompting Visual Coherence Prompt-based spatial reasoning, no edit ops. SMARTEdit-Bench (Ours) Posters, Web, Images ✓ ✓ ✓ Implicit instructions requiring Cascaded-Reasoning Visual, Semantic, Structural Multi-domain, layout propagation, semantic + visual edit cascades. Table 1: Comparison of SMARTEdit-Bench with prior datasets. Our benchmark uniquely supports multi-domain, instruction-grounded cascading edits with compositional reasoning. Related works are described in Appendix 10 . Layout and scene editing is a cornerstone of multimodal reasoning, encompassing tasks such as poster restructuring, webpage reflow, and natural image transformation Suri et al. 2024 ; Zhu et al. 2024 ; Gani et al. 2024 ; Tanaka et al. 2024 ; Lin et al. 2024 . These edits often go beyond simple object manipulations and require coordinated adjustments across spatial, semantic, and stylistic dimensions Jia et al. 2024 ; Feng et al. 2023a ; Lin et al. 2023 . Instructions like “Insert a video section at the middle” (Figure 1 ) require cascading edits such as identifying an insertion point in layout hierarchy, shifting subsequent sections to preserve flow, aligning with adjacent regions, and updating labels and narrative rhythm for coherence. When humans learn how to draw or typset, they learn about the basics of composition and balance Golomb 1987 , but it remains unclear whether large language models (LLMs) or visual LMs (VLMs) can propagate edits in a structured and semantically coherent manner. These tasks are critical for scientific communication Lozano 2014 , where even a single misplaced section or inconsistent scene can undermine clarity, or usability. Figure 1: Unlike InstructPix2Pix Brooks et al. 2023 , HIVE Zhang et al. 2024b , and Base-LLama, which apply edits locally and often fail to maintain coherence, SMART-Editor performs structured layout edits and visual transformations with compositional reasoning—anticipating cascading effects and preserving global alignment and semantic consistency. To explore this challenge, we introduce the notion of a SMART-EDIT —an edit that (a) adheres semantically to the instruction, (b) minimizes disruption to unrelated content, (c) preserves spatial coherence and alignment, and (d) anticipates and resolves cascading effects (formalized in Section 2 ). We further present SMARTEdit-Bench (Section 3 ), a benchmark designed to evaluate compositional edit reasoning in LLMs and VLMs across three domains: scientific posters, web layouts, and natural images. Each instance in the benchmark features a real-world or curated edit sourced from datasets such as SciPostLayout Tanaka et al. 2024 , Design2Code Si et al. 2025 , and HQEdit Hui et al. 2024 , comprising an initial scene, a human-verified SMART-EDIT instruction, and a target layout/image requiring multi-step semantic and structural updates. Unlike prior datasets focused on isolated edits (Table 1 ), SMARTEdit-Bench is designed to evaluate the ability of models to perform globally coherent, multi-step edits to maintain human-like design integrity . We observe that base LLMs/VLMs like LLaMA, Qwen, LLaVA, GPT4-o and Gemini-pro can follow instructions superficially (edit adherence >= 4.0), but fail to preserve deeper layout reasoning: narrative coherence drops by over 15% during section reordering, and layout alignment degrades sharply during insert/delete operations (Table 2 , 3 , 4 ). Cross-sectional consistency is especially fragile when instructions induce global changes (e.g., regrouping content or propagating section deletions). To address these limitations, we introduce SMART-Editor , a modular agentic framework that coordinates planning, execution, critique, and optimization (Section 4 and Figure 2 ). The Action Agent translates edit instructions into layout operations, which are applied by the Executor . A Critique Agent then evaluates the output across structured reward axes (e.g., overlap, whitespace, coherence) and produces actionable feedback in natural language. This feedback/new action plan guides the Optimizer Agent , which operates in two modes: (1) Reward-Refine , which uses critiques to revise the initial plan in an iterative loop until all quality checks are satisfied; and (2) RewardDPO , which fine-tunes the model using preference pairs filtered by reward alignment to generalize better layout reasoning policies. Across posters, webpages, and natural images, SMART-Editor improves layout alignment (+0.3–0.5), narrative and cross-sectional consistency (up to +0.2), and visual plausibility (Figure 3 , 4 ). 2 Evaluation of ‘SMARTness’ of an Edit? Inspired by the evaluation metrics used by prior layout generation works Tanaka et al. 2024 ; Lin et al. 2023 ; Hui et al. 2024 ; Zhang et al. 2024b , we formalize the notion of SMART-Edit as an edit operation that, when applied to structured or unstructured visual content, satisfies a set of post-edit quality constraints in structured domain ((Definitions and Mathematical Formulations in Appendix 12.1 ) such as 1) Edit Adherence verifies whether the edit instruction is faithfully executed 2) Narrative Coherence ensures logical section ordering (e.g., Methods before Results ) is preserved. Violations are detected by prompting GPT-4o on expected order pairs 3) Cross-Sectional Consistency evaluates whether semantically linked elements (e.g., captions and figures, objects and context) remain aligned. Contradictions are flagged via entailment-based GPT-4o prompts 4) Visual-Spatial Layout captures design quality through three components: overlap penalty (discouraging collisions), whitespace penalty (encouraging compactness), and alignment reward (favoring grid-aligned layouts). However, in unstructured natural images , we formulate four core capabilities (Definitions and Mathematical Formulations in Appendix 12.3 ) 1) Edit Adherence (EA) - Same as before, 2) Semantic match (SM) : Compute the CLIP similarity between the textual instruction and the generated image, 3) Object Size and Realism (Overlap) : Check if the edited image contains objects that appear too large, too small, or overlap unnaturally, 4) Depth Layering and Occlusion (DL)

Read the complete text