Instance Segmentation of Scene Sketches Using Natural Image Priors

Mia Tang, Yael Vinker, Chuan Yan, Lvmin Zhang, Maneesh Agrawala · 2025

Read Instance Segmentation of Scene Sketches Using Natural Image Priors by Mia Tang, Yael Vinker, Chuan Yan, Lvmin Zhang, Maneesh Agrawala (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-09608.txt.

Introduction

Mia Tang and co-authors present a method to split a scene sketch into its single objects in 2025. Splitting helps edit tasks like moving or resizing one part. Photo tools do this well, but sketches are sparse and vary in style. Their method, called InkLayer, adapts photo tools to sketches.

Fine tuning that ignores class labels is the first step. Depth cues then refine the masks for each object. Found objects are sorted into layers. Hidden parts of blocked objects are filled in, which aids deep edits. The team also built a fake data set with many stroke styles and detail levels. Tests on that set show the method holds up across styles.

Catalog record

CREATORMia Tang, Yael Vinker, Chuan Yan, Lvmin Zhang, Maneesh Agrawala
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-09608

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Instance Segmentation of Scene Sketches Using Natural Image Priors Mia Tang, Yael Vinker, Chuan Yan, Lvmin Zhang, Maneesh Agrawala Abstract . Sketch segmentation involves grouping pixels within a sketch that belong to the same object or instance. It serves as a valuable tool for sketch editing tasks, such as moving, scaling, or removing specific components. While image segmentation models have demonstrated remarkable capabilities in recent years, sketches present unique challenges for these models due to their sparse nature and wide variation in styles. We introduce InkLayer , a method for instance segmentation of raster scene sketches. Our approach adapts state-of-the-art image segmentation and object detection models to the sketch domain by employing class-agnostic fine-tuning and refining segmentation masks using depth cues. Furthermore, our method organizes sketches into sorted layers, where occluded instances are inpainted, enabling advanced sketch editing applications. As existing datasets in this domain lack variation in sketch styles, we construct a synthetic scene sketch segmentation dataset, InkScenes , featuring sketches with diverse brush strokes and varying levels of detail. We use this dataset to demonstrate the robustness of our approach. Code and data for this paper are released at project page: https://inklayer.github.io . 224 Conference: Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers ; August 10–14, 2025; Vancouver, BC, Canada Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers (SIGGRAPH Conference Papers ’25), August 10–14, 2025, Vancouver, BC, Canada DOI: 10.1145/3721238.3730606 ISBN: 979-8-4007-1540-2/2025/08 CCS: Computing methodologies Image segmentation CCS: Computing methodologies Non-photorealistic rendering CCS: Computing methodologies Scene understanding Figure 1. Our method performs instance segmentation of raster sketches. It effectively handles diverse types of sketches, accommodating variations in stroke style and complexity. 1. Introduction Sketches serve as a powerful tool for visual exploration, ideation, and planning. Traditional sketching workflows often begin with artists working on a single canvas layer (either physical or digital) to maintain creative momentum. As the sketch evolves and requires refinement (e.g., adjustments to composition, perspective, or other elements), artists face the tedious task of manually segmenting different elements of the sketch into discrete, editable layers. Automating the sketch segmentation process offers a promising solution. However, this task presents unique challenges due to the sparse and abstract nature of line drawings, as well as the inherent variability in human sketching styles. Existing methods for scene-level sketch segmentation typically rely on training dedicated models using annotated sketch datasets ( Sangkloy et al., 2016 ; Zou et al., 2018 ; Chowdhury et al., 2022 ) . However, most available datasets are confined to specific sketch styles and a limited set of object categories, restricting the generalization capabilities of existing methods. In this work, we introduce InkLayer , a method for instance segmentation of raster scene sketches that outperforms previous approaches in accommodating a wider variety of sketch styles and concepts. We use the segmentation map to divide the sketch into sorted layers to support effective sketch editing. Our method builds upon Grounded SAM ( Ren et al., 2024 ) , a state-of-the-art approach for open-vocabulary image segmentation, which has demonstrated remarkable capabilities in segmenting complex scenes across diverse object categories. Grounded SAM combines two models to achieve this: Grounding DINO ( Liu et al., 2023b ) for object detection and Segment Anything (SAM) ( Kirillov et al., 2023 ) for mask generation. We analyze the performance of these models on sketch, revealing that the domain gap between real and sketched objects presents significant challenges for Grounding DINO. In contrast, SAM exhibits a surprising ability to generalize to sketches, though it still faces difficulties specific to the sketch domain. To address the gap in object detection, we fine-tune Grounding DINO on a small subset of annotated scene sketches from the SketchyScene dataset ( 2018 ) . Our fine-tuning technique achieves a substantial improvement in Grounding DINO’s detection performance on sketches, with Average Precision increasing from 26% to 74% . For object segmentation, we apply SAM ( Kirillov et al., 2023 ) in the sketch domain, using detected object regions from our finetuned Grounding DINO. This is followed by a depth-based refinement stage to resolve ambiguities in overlapping regions. Finally, we decompose the segmented sketch into sorted layers and employ a pretrained image inpainting model ( 2022 ) to fill in missing regions. This layered representation facilitates sketch editing, allowing users to drag or manipulate segmented objects without the need to manually sketch the affected regions, as we demonstrate in the provided video. To evaluate our method on diverse scene sketches, we construct InkScenes , a synthetic annotated dataset of sketched scenes that extends existing benchmarks along three key dimensions: drawing style, stroke style, and object categories. The dataset integrates two complementary pipelines to enhance diversity. The first pipeline builds on SketchyScene ( Zou et al., 2018 ) , expanding its clipart-like sketches with styles ranging from high-fidelity representations to symbolic, abstract sketches, introducing challenging out-of-distribution cases. The scenes are created in vector format to allow for stroke style variations, including Calligraphic Pen, Charcoal, and Brush Pen styles. The second pipeline leverages the Visual Genome dataset ( Krishna et al., 2017 ) , which provides annotated scenes with object variety. Using the InstantStyle method ( Wang et al., 2024 ) , we generate expressive, natural-looking sketches spanning 72 categories, extending SketchyScene’s original 45 categories by 53 new categories. Our dataset contains 20,542 annotated scene sketches in total, and is highly extensible. Our evaluations demonstrate that InkLayer generalizes well to these challenging variations, significantly advancing the state of the art. Figure 2. Overview of the sketch segmentation pipeline. Given an input sketch image, our framework first detects bounding boxes using a customized Grounding DINO to obtain region proposals, and then perform segmentation with SAM models. The localization and segmentation are refined by incorporating the depth features. The result segmentation can be viewed as a layered decomposition of object components in the original sketch. 2. Related Work 2.1. Part-Level Sketch Segmentation The majority of work in the sketch segmentation domain focuses on part-level semantic segmentation, in which the goal is to assign labels to object parts ( e.g . , the body, wings, and head of a bird). These methods often rely on curated part-level sketch segmentation datasets ( Ge et al., 2020 ; Li et al., 2018 ; Wu et al., 2018 ; Eitz et al., 2012 ; Huang et al., 2014 ) to train a segmentation model, and use various network architectures, including CNNs ( Zhu et al., 2018 ; Wang et al., 2020 ) , RNNs ( Qi and Tan, 2019 ; Wu et al., 2018 ; Kaiyrbekov and Sezgin, 2020 ) , Graph Neural Networks ( Zheng et al., 2022 ; Yang et al., 2021 ) , Transformers ( Wang and Li, 2024 ; Zheng et al., 2023 ) , and more specific techniques such as deformation networks ( Qi et al., 2021 ) and CRFs ( Schneider and Tuytelaars, 2016 ) . These approaches typically operate on a fixed set of object classes, and recognize a predefined set of object parts within them. Other work focuses on perceptual grouping ( Li et al., 2018 ; Li et al., 2019 ; Li et al., 2022 ) to achieve class-agnostic segmentation.

Read the complete text