SpotEdit: Evaluating Visually-Guided Image Editing Methods

Sara Ghazanfari, Wei-An Lin, Haitong Tian, Ersin Yumer · 2025

Read SpotEdit: Evaluating Visually-Guided Image Editing Methods by Sara Ghazanfari, Wei-An Lin, Haitong Tian, Ersin Yumer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18159.txt.

Introduction

Sara Ghazanfari, Wei-An Lin, Haitong Tian, and Ersin Yumer present SpotEdit in two thousand twenty-five. SpotEdit benchmarks image editing guided by both sight and text. Such edits promise fine, steerable control of generated content. Past tests were too simple to mirror real editing demands.

The benchmark spans diffusion, autoregressive, and hybrid generators. Scores split widely across those model families. A special part probes hallucination in the editing step. Top models often invent a visual cue that is not there. They then edit as if the cue were real. Code and data are shared in public.

Catalog record

CREATORSara Ghazanfari, Wei-An Lin, Haitong Tian, Ersin Yumer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-18159

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SpotEdit: Evaluating Visually-Guided Image Editing Methods Sara Ghazanfari, Wei-An Lin, Haitong Tian, Ersin Yumer Abstract Visually-guided image editing, where edits are conditioned on both visual cues and textual prompts, has emerged as a powerful paradigm for fine-grained, controllable content generation. Although recent generative models have shown remarkable capabilities, existing evaluations remain simple and insufficiently representative of real-world editing challenges. We present SpotEdit, a comprehensive benchmark designed to systematically assess visually-guided image editing methods across diverse diffusion, autoregressive, and hybrid generative models, uncovering substantial performance disparities. To address a critical yet underexplored challenge, our benchmark includes a dedicated component on hallucination , highlighting how leading models, such as GPT-4o, often hallucinate the existence of a visual cue and erroneously perform the editing task. Our code and benchmark are publicly released at github.com/SaraGhazanfari/SpotEdit . 1 Introduction † † footnotetext: ∗ Work done during internships at Adobe Inc. Visually-guided image editing enables precise, localized manipulation by combining a reference image with textual instructions to guide generative models. Compared to text-only editing, this multimodal approach provides greater control, stronger semantic alignment, and higher spatial precision, making it valuable for applications such as consistent keyframe editing in the area of long-form video editing [ 27 , 5 ] . Despite rapid advances in diffusion-based [ 20 , 22 ] and autoregressive [ 3 , 16 ] generative methods, rigorous evaluation of visually guided editing remains underexplored. Existing benchmarks [ 23 , 7 ] focus largely on coarse manipulations, simple object replacements, or single-object scenes (see Fig. 4 ), offering limited insight into the complexities of real-world editing. As a result, current evaluations fail to capture the nuanced challenges of multimodal guidance, hindering fair model comparison and progress. To close this gap, we introduce SpotEdit, a comprehensive benchmark for systematic and fine-grained evaluation of visually guided image editing. SpotEdit is built from diverse real and synthetic video frames, enabling controlled variation in object appearance, position, scale, and context. More specifically, each benchmark instance consists of: a reference image, an input image, and a textual instruction, and a ground-truth target image. Crucially, SpotEdit also includes a dedicated Hallucination subset that probes failure cases where objects are missing from either the reference or the input image. This component directly evaluates a model’s robustness to edge cases, testing its ability to avoid spurious insertions while preserving both spatial coherence and semantic fidelity under adverse conditions. We evaluate leading open- and closed-source models, including OmniGen2 [ 26 ] , BAGEL [ 3 ] and GPT-4o [ 12 ] on SpotEdit. Results reveal that visually guided editing remains fundamentally challenging: the strongest open-source model achieves only 0.685 similarity score to ground truth. Moreover, models exhibit complementary strengths and weaknesses, e.g., OmniGen2 adheres closely to visual guidance but disrupts background consistency, while BAGEL preserves context yet struggles with cue interpretation. Strikingly, in hallucination cases, even the proprietary GPT-4o performs poorly, hallucinating object presence and executing incorrect edits despite its strong general image editing capabilities. 2 Related Work In the following, we present related work on existing visually-guided image editing benchmarks. An extended discussion, including related work on the methods, is provided in App. A . Visually-guided image editing benchmarks. As a pioneer, Paint by Example [ 23 ] introduced exemplar-based editing by inpainting image regions from a reference exemplar, focusing on visual similarity and identity preservation. DreamEdit [ 7 ] extended this to subject-driven editing, manipulating a subject’s appearance or context while preserving identity. As shown in Fig. 4 , prior benchmarks involve simple scenes, few distractors, and near-identical object poses across reference and edited images. In contrast, SpotEdit targets fine-grained, visually-guided editing in complex scenes with multiple objects, sparse textual descriptions, challenging spatial layouts, and pose variations. Its deterministic design with a ground truth enables objective evaluation and, for the first time, tests hallucination to incomplete or missing visual cues. 3 SpotEdit Benchmark In this section, we outline our data generation pipeline, present benchmark statistics, and highlight key characteristics. Additional details are provided in App. B . 3.1 Data Generation Pipeline We construct the SpotEdit benchmark through a structured data generation pipeline that ingests keyframes extracted from video sources. Our data comes from two complementary datasets: (1) the StoryStream dataset [ 24 ] , a large-scale, high-resolution synthetic multimodal collection designed for long-form story generation, and (2) real (non-synthetic) videos from NExT-QA [ 21 ] , which add diversity and visual realism. While StoryStream provides keyframes directly, NExT-QA requires preprocessing to extract them, details of which are given in App. B . Starting with the key-frames, we then utilize our three-stage data generation process, illustrated in Fig. 1 and described below, to generate our benchmark: Step 1: Instruction Generation. Both datasets include frame-level captions, which we use to prompt Llama-3.1-8B-Instruct to produce fine-grained instructions for object-level image editing. The model outputs an editing prompt, the target object, and the corresponding frames. This stage is entirely text-based, relying solely on the narrative captions; no visual inputs are required. Step 2: Frame Localization. To refine the Location of the target object, i.e., specific frames where the target object appears, we query InternVL3-8B [ 29 ] . Each frame is individually evaluated by the model to determine the presence or absence of the target object. Step 3: Consistent Editing. Once the target frames are identified, we perform image editing using the GPT-4o model. More specifically, the first image is edited without any visual guidance. While other frames are edited using the edited version of the first frame as the visual guidance to preserve edit consistency over all keyframes. We construct benchmark samples by pairing each edit instruction with its corresponding source and edited images. In the standard setting, each sample consists of: a reference frame (providing visual guidance), a source frame (the image to be edited), a textual instruction, and a ground-truth edited frame (the target output). In the hallucination setting, we deliberately introduce cases where the target object is absent. Specifically, Ref. Robustness samples lack the target object from the reference image, while Inp. Robustness samples lack it from the source image. Figure 1 : SpotEdit data generation pipeline. The pipeline consists of three key stages: (1) generating editing instructions from frame-wise video descriptions, (2) identifying target frames containing the specified object using multimodal queries, and (3) applying consistent edits only to the relevant frames utilizing Visually-guided image editing. 3.2 Benchmark Statistics & Key Features Our benchmark contains 500 samples spanning both synthetic and real images. As illustrated in Fig. 7 , approximately 40% of the samples belong to the hallucination setting, while the remaining 60% correspond to the standard setting. SpotEdit introduces several distinctive features. First, it explicitly evaluates hallucination by including cases where the source object is absent, requiring models to correctly refrain from editing. Second, instead of isolated images, it leverages video keyframes that capture diverse object poses, scales, and lighting conditions, making the task more realistic and challenging. Third, editing instructions are intentionally concise, pushing models to rely on visual grounding for consistency with the reference image. Finally, each task is paired with a nearly unique ground-truth target, allowing precise and unambiguous evaluation, unlike traditional benchmarks. Together, these design choices enable systematic assessment of both standard editing performance and

Read the complete text