ScribbleEdit: Synthetic Data for Image Editing with Scribbles and Text
Anya Ji, George Ma, Téa Wright, Yiming Zhang, Alane Suhr, Somayeh Sojoudi · 2026
Read ScribbleEdit: Synthetic Data for Image Editing with Scribbles and Text by Anya Ji, George Ma, Téa Wright, Yiming Zhang, Alane Suhr, Somayeh Sojoudi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01135.txt.
Introduction
Anya Ji and six co-authors introduce ScribbleEdit, a synthetic dataset for image editing that combines freehand scribbles with text instructions. Text conveys semantics such as texture and color but lacks spatial precision, while scribbles give rough layout but cannot express detailed attributes. Existing models struggle to interpret both together because specialized training data was missing.
The dataset is built by a pipeline that generates source-target image pairs through inpainting, then pairs them with human-drawn scribbles and model-generated text instructions. Off-the-shelf diffusion and autoregressive editing models handle abstract scribbles poorly. Finetuning on ScribbleEdit significantly improves their ability to produce edits that match in both layout and meaning.
Catalog record
| CREATOR | Anya Ji, George Ma, Téa Wright, Yiming Zhang, Alane Suhr, Somayeh Sojoudi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.01135
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ScribbleEdit: Synthetic Data for Image Editing with Scribbles and Text Anya Ji, George Ma, Téa Wright, Yiming Zhang, Alane Suhr, Somayeh Sojoudi Abstract Recent progress in generative models has significantly advanced image editing capabilities, yet precise and intuitive user control remains difficult. Specifically, users often struggle to communicate both exact spatial layouts and specific semantic details simultaneously. While natural language instructions effectively convey high-level semantics like texture and color, they lack spatial specificity. Conversely, freehand scribbles provide rough spatial boundaries but cannot express detailed visual attributes. Consequently, achieving precise control requires combining both modalities. However, existing models struggle to jointly interpret abstract scribbles alongside text due to a lack of specialized training data. In this work, we introduce ScribbleEdit , a large-scale synthetic dataset designed to bridge this gap by combining natural language instructions with freehand scribble inputs for more accurate, controllable edits. We construct this dataset through a synthetic pipeline that automatically generates source-target image pairs via inpainting, which are then paired with human-drawn scribbles and VLM-generated text instructions. Using ScribbleEdit, we evaluate and finetune both diffusion-based and autoregressive unified multimodal image editing models. Our experiments reveal that while off-the-shelf models struggle with abstract scribble inputs, finetuning on our synthetic dataset significantly improves their ability to generate spatially aligned and semantically consistent edits. 1 Introduction and Background Despite progress in image editing algorithms, we are still unable to achieve precise control over edits, primarily due to the limitations of natural language. While language instructions are expressive for describing high-level semantics, they often lack spatial specificity, making it difficult to control where and how edits appear. Freehand scribbles provide a complementary solution, offering an intuitive control for users to quickly outline an edit’s location and structure. However, because scribbles are inherently abstract and imprecise representations, they are significantly harder for models to interpret and translate into accurate edits. Figure 1 : Overview of the ScribbleEdit pipeline for dataset construction and model finetuning. Mask-based inpainting is a common technique for generating training data for image editing by removing objects and generating textual instructions for reconstruction ( 20 ; 1 ) . However, these approaches typically focus on text-only editing and provide limited spatial guidance. Scribbles offer a lightweight visual signal that captures rough spatial structure while leaving appearance details to the generative model. Unlike precise masks or detailed sketches, scribbles are abstract and imprecise, making them challenging for existing models to interpret reliably. At the same time, they are intuitive and efficient for users to produce, requiring only minimal effort to indicate approximate shape and placement. This makes scribbles a practical interface for controllable image editing while posing a challenge for generative models. Prior work has explored sketch-guided editing ( 23 ; 14 ; 9 ) , as well as multimodal conditioning for image generation using text and visual inputs such as sketches, edges, or masks ( 24 ; 10 ; 5 ) . For image editing, SmartBrush 21 conditions object inpainting on both text and shape for semantic and spatial guidance. ReEdit ( 16 ) further reduces ambiguity by incorporating exemplar images alongside text. Interactive systems such as MagicQuill ( 8 ) further demonstrate the potential of combining multiple visual cues for controllable editing. However, datasets designed specifically for studying scribble- and text-guided image editing remain limited. To address this gap, we introduce ScribbleEdit , a large-scale synthetic dataset for multimodal image editing that combines scribbles and natural language instructions. Our pipeline constructs paired examples by removing objects from images in the Sketchy Database ( 13 ) and generating instructions for restoring them, as illustrated in Figure 1 . We evaluate the dataset by training and benchmarking diffusion-based and unified multimodal models under different conditioning settings, including text-only, scribble-only, and joint scribble–text guidance. Our results show that scribble-guided editing remains challenging for current models and finetuning on ScribbleEdit effectively improves models’ ability to generate spatially accurate and semantically consistent images based on scribble and text. Overall, our contributions are: (1) ScribbleEdit, a dataset for scribble- and text-guided image editing; (2) an automated pipeline for constructing multimodal editing examples; and (3) a benchmark evaluation of modern image editing models under different conditioning settings. 2 Dataset Generation We construct a dataset of 11,775 samples for the object addition task, where the goal is to add a specific object to an image guided by a scribble and a natural language instruction. Each sample consists of four components: an input image with the object removed, the original photo serving as the target image , a hand-drawn scribble of the object, and a natural language instruction describing the edit. Source data. We subsample 12,500 image-scribble pairs from the Sketchy Database ( 13 ) , a large-scale collection of photos paired with hand-drawn scribbles contributed by crowd workers. Each photo depicts a single object from one of 125 categories; every photo is accompanied by between 5 and 18 scribbles (6 on average), each drawn by a different annotator. We retain samples in which the object occupies a reasonable portion of the image and the scribbles visually correspond to the object’s position in the photo. Object segmentation. To obtain object masks, we apply a two-stage pipeline. First, grounding DINO ( 7 ) detects the object using the class name as the text prompt to produce a bounding box. Second, the box is passed to the Segment Anything Model (SAM) ( 6 ) (ViT-H variant), which produces three mask candidates. We select the candidate with the highest predicted IoU score. The selected mask is dialted by 15 pixels using a 15\times 15 structuring element to ensure complete coverage of the object boundary. Input image generation. We remove the segmented object from each photo using the LaMa inpainting model ( 17 ) , which fills the masked region with a plausible background. Because the mask already identifies the object precisely, no text prompt is needed; LaMa reconstructs the background purely from the surrounding image context. The resulting inpainted image serves as the input to the editing model. Of the 12,500 sampled photos, 11,775 yield a valid input image after filtering out cases where object detection or segmentation fails. Instruction generation. We generate a natural language instruction for each sample using Gemma 3 (4B) ( 18 ) , an instruction-tuned vision-language model. The model is given the input image and the target image together with a prompt that specifies the editing operation (e.g., “Add a class to the image.”) and asks the model to produce a single, specific editing command that describes the object’s appearance and location. Dataset splits. We split the 11,775 samples into training, validation, and test sets containing 9,438, 1,164, and 1,173 unique photos, respectively. Since each photo is paired with multiple scribbles, to reduce training cost, we sample a subset in which each photo is paired with at most two scribbles, giving 18,876 training, 2,328 validation, and 2,346 test instances. An example is shown in Figure 1 . Model Inputs Histogram Intersection \uparrow RMSE \downarrow CLIP \uparrow FID \downarrow Entire BBOX Entire BBOX Entire Entire SD Inpaint + ControlNet-Scribble 24 S+T+M 0.82 0.49 36.10 78.76 0.81 53.63 BAGEL-7B-MoT 2 S+T 0.63 0.53 58.13 71.55 0.68 112.92 Image-Cond. ControlNet-Scribble (finetuned) T 0.68 0.49 56.94 82.12 0.87 28.90