FocalClick-XL: Towards Unified and High-quality Interactive Segmentation
Xi Chen, Hengshuang Zhao · 2025
Read FocalClick-XL: Towards Unified and High-quality Interactive Segmentation by Xi Chen, Hengshuang Zhao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-14686.txt.
Introduction
Xi Chen and Hengshuang Zhao present FocalClick-XL, a system for interactive segmentation, in work from 2025. Users mark target objects with clicks, scribbles or boxes, and the system returns a mask. Older methods often accept few input forms and miss fine detail, so the authors revisit a classical coarse-to-fine design.
The task is split into meta-tasks for context, object and detail, each with its own subnet and scaled pre-training. A prompting layer at the object level encodes the input type, while context and detail are shared. Results set a new high on click-based tests and adapt well to boxes, scribbles and coarse masks. The system can also predict alpha mattes.
Catalog record
| CREATOR | Xi Chen, Hengshuang Zhao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-14686
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FocalClick-XL: Towards Unified and High-quality Interactive Segmentation Xi Chen, Hengshuang Zhao Abstract Interactive segmentation enables users to extract binary masks of target objects through simple interactions such as clicks, scribbles, and boxes. However, existing methods often support only limited interaction forms and struggle to capture fine details. In this paper, we revisit the classical coarse-to-fine design of FocalClick and introduce significant extensions. Inspired by its multi-stage strategy, we propose a novel pipeline, FocalClick-XL , to address these challenges simultaneously. Following the emerging trend of large-scale pretraining, we decompose interactive segmentation into meta-tasks that capture different levels of information—context, object, and detail—assigning a dedicated subnet to each level. This decomposition allows each subnet to undergo scaled pretraining with independent data and supervision, maximizing its effectiveness. To enhance flexibility, we share context- and detail-level information across different interaction forms as common knowledge while introducing a prompting layer at the object level to encode specific interaction types. As a result, FocalClick-XL achieves state-of-the-art performance on click-based benchmarks and demonstrates remarkable adaptability to diverse interaction formats, including boxes, scribbles, and coarse masks. Beyond binary mask generation, it is also capable of predicting alpha mattes with fine-grained details, making it a versatile and powerful tool for interactive segmentation. I Introduction Interactive image segmentation requires users to indicate the target by providing simple indications such as boxes [ 1 , 2 ] , scribbles [ 3 , 4 , 5 , 6 ] , and clicks [ 7 , 8 , 9 , 10 ] . Compared with traditional annotation tools like the lasso or brush, interactive models could largely reduce the cost of creating masks, which is important for efficient annotation in the era of big data. Initial works for interactive segmentation [ 8 , 9 , 7 ] mainly focus on designing delicate model structures or developing better strategies [ 11 , 12 ] for simulating user interactions. However, training on limited data constrains the power of those solutions as the segmentation target could be “anything”. Recently, Segment Anything Model (SAM) [ 2 ] tackles interactive segmentation with a data-centric solution. It iteratively collects masks with humans in the loop, and finally collects billions of high-quality samples and trains a powerful segmentation model. The generalized strong priors make SAM a foundation solution for interactive segmentation and downstream applications. However, SAM is not a perfect solution for interactive segmentation as it makes some compromises for “automatic segmentation”. First, SAM treats each part of the full image equally without any design of object-centric modeling or detail refinements, which are proven crucial for the fine details in previous interactive segmentation works [ 9 , 13 , 12 , 14 ] . The compromise is that SAM considers the computation burden for each independent interaction as it requires densely sampled points for automatic segmentation. Thus, SAM develops a huge encoder to extract the common features for each part of the image and leaves a small prompt encoder to model each interaction. Besides, SAM only supports click and box, and it is hard for the current prompt encoder to encode more flexible user interactions like scribbles and coarse masks 1 1 1 SAM supports the previous/initial masks as inputs, but shows very limited improvements w/o additional click or box. . Different from SAM, we do not consider automatic segmentation but focus on building a state-of-the-art interactive segmentation system. Fig. 1 : Demonstrations for FocalClick-XL . Our method is compatible with various formats of user interactions like clicks, scribbles, boxes, coarse masks, etc. , and could predict highly-refined details for both transparent and solid objects. Among previous interactive segmentation methods, FocalClick [ 9 ] designs an efficient pipeline following the coarse-to-fine fashion. In this work, we make extensions based on this classical framework and propose FocalClick-XL for high-quality and unified interactive segmentation. Our main insight is to decompose the interactive segmentation pipeline into different subnets to focus on context, object, and detail-level information. This modular design brings the following advantages. First, each subnet could be pre-trained sufficiently with large-scale specific data, which assists our FocalClick-XL for high-quality segmentation. Specifically, we leverage the pre-trained visual encoder of SAM for the Context-Net, which takes the full image as input for capturing the global context. The Object-Net takes the image patch around the segmentation target and produces the primitive segmentation masks according to the provided user interactions. This part is trained on a combination dataset with object-centric images. The Detail-Net takes charge of refining the predicted details. It takes the small image patches around the boundaries or low-confident regions as input and is trained on a combined dataset of image segmentation and matting samples. Second, in this decomposed pipeline, only the Object-Net is interaction-sensitive, thus we could share the majority of knowledge ( i.e. , Context-Net and Detail-Net) across different interactions. It significantly eases the goal of building a unified framework for various interaction forms. Concretely, we add a single Prompting Layer at the input of the Object-Net to encode different types of interactions. To support novel interaction forms, we only need to train a novel Prompting Layer and keep other parameters frozen. To mitigate the computation burden brought by the cascaded subnet, we follow the strategy of FocalClick. During inference, the Context-Net runs only once per image. After each round of user interaction, we dynamically zoom in on the object patch and detail patch with a small input size to accelerate the Object-Net and Detail-Net. Besides the exploration of the model structure, we thoroughly investigate how to evaluate each type of interaction and propose a series of evaluation protocols and benchmarks. For example, we developed a deterministic scribble generator that supports evaluating scribble-based methods automatically. Besides, we construct a series of coarse mask sets with different qualities to evaluate the robustness for the task of coarse mask refinement. As presented in Fig. 1 . FocalClick-XL is compatible with various of user interactions and could predict high-quality masks with fine details. Extensive experiments show that FocalClick-XL demonstrates state-of-the-art performance on click-based interactive segmentation benchmarks, and shows competitive performances for various interaction formats. In general, the contribution of this work could be summarized in three folds: First, our work makes FocalClick [ 9 ] benefit from large-scale training, thus reaching state-of-the-art performance for click-based segmentation benchmarks. Second, we design a decomposed pipeline that extends FocalClick for novel interaction forms like scribbles, boxes, and coarse masks. Third, we carry out a thorough study of various interactions and formulate well-defined train/val protocols and benchmarks, which would benefit the community in exploring more types of interaction. Difference from conference version. This manuscript improves the conference version [ 9 ] significantly with wider extensions for up-scaled training and support for more interactions. 1) In Section IV , we up-scaled the model design and training data for each stage of FocalClick and apply different training strategies in each stage. 2) We extend the user-interaction support for FocalClick from click to wider types, including scribble, box and coarse masks in Section V . At the same time, we thoroughly investigate the evaluation protocol for these new interactions. 3) We give a more detailed experimental analysis for the newly added part and scaling strategies in Section VII , and add more