X2SAM: Any Segmentation in Images and Videos

Hao Wang, Limeng Qiao, Chi Zhang, Lin Ma, Guanglu Wan, Xiangyuan Lan, Xiaodan Liang · 2026

Read X2SAM: Any Segmentation in Images and Videos by Hao Wang, Limeng Qiao, Chi Zhang, Lin Ma, Guanglu Wan, Xiangyuan Lan, Xiaodan Liang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00891.txt.

Introduction

Hao Wang and six co-authors introduce X2SAM, a unified model that brings language-guided segmentation to both images and videos. Multimodal language models understand images at a broad level, but pixel-level perception lags. Foundation segmenters make high-quality masks yet cannot interpret complex conversational instructions, and earlier language segmenters usually handled only images or only videos, with limited prompt types.

X2SAM couples a language model with a Mask Memory module that stores guided vision features for consistent masks over time. One formulation supports generic, open-vocabulary, referring, reasoning, interactive, and grounded tasks for both media. A new video benchmark tests segmentation from interactive visual prompts. Joint training keeps strong segmentation while preserving general chat ability.

Catalog record

CREATORHao Wang, Limeng Qiao, Chi Zhang, Lin Ma, Guanglu Wan, Xiangyuan Lan, Xiaodan Liang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.00891

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

X2SAM: Any Segmentation in Images and Videos Hao Wang, Limeng Qiao, Chi Zhang, Lin Ma, Guanglu Wan, Xiangyuan Lan, Xiaodan Liang Abstract Multimodal Large Language Models (MLLMs) have demonstrated strong image-level visual understanding and reasoning, yet their pixel-level perception across both images and videos remains limited. Foundation segmentation models such as the SAM series produce high-quality masks, but they rely on low-level visual prompts and cannot natively interpret complex conversational instructions. Existing segmentation MLLMs narrow this gap, but are usually specialized for either images or videos and rarely support both textual and visual prompts in one interface. We introduce X2SAM, a unified segmentation MLLM that extends any-segmentation capabilities from images to videos. Given conversational instructions and visual prompts, X2SAM couples an LLM with a Mask Memory module that stores guided vision features for temporally consistent video mask generation. The same formulation supports generic, open-vocabulary, referring, reasoning, grounded conversation generation, interactive, and visual grounded segmentation across image and video inputs. We further introduce the Video Visual Grounded (V-VGD) segmentation benchmark, which evaluates whether a model can segment object tracks in videos from interactive visual prompts. With a unified joint training strategy over heterogeneous image and video datasets, X2SAM delivers strong video segmentation performance, remains competitive on image segmentation benchmarks, and preserves general image and video chat ability. 1]Sun Yat-Sen University 2]Peng Cheng Laboratory 3]Meituan Inc \contribution [†]Corresponding author \code https://github.com/wanghao9610/X2SAM \project https://wanghao9610.github.io/X2SAM \correspondence ,, 1 Introduction Figure 1 : Comprehensive capabilities of X2SAM. Guided by conversational instructions, X2SAM performs diverse Image and Video segmentation tasks. X2SAM provides a unified interface for a wide range of image and video segmentation tasks, including generic, referring, reasoning, grounded conversation generation, interactive, visual grounded, object-centric, and out-of-domain segmentation. This design highlights how image-level any-segmentation capabilities can be extended to video inputs with textual and visual prompts. Multi-modal Large Language Models (MLLMs) have exhibited substantial advancements alongside the rapid development of Large Language Models (LLMs) [ 3 , 51 ] and multi-modal pre-training methods [ 46 , 15 ] . These models have shown remarkable effectiveness in a wide range of applications, including image captioning [ 60 ] , VQA [ 1 ] , and visual editing [ 7 ] . However, while current MLLMs excel at global visual understanding, their capability to generate dense, pixel-level outputs for precise spatial and temporal comprehension remains limited. This limitation poses a considerable challenge in directly addressing fine-grained tasks across both static images and dynamic video sequences. Foundation segmentation models, such as SAM [ 18 ] and its video-extended successor SAM2 [ 49 ] , generate dense masks across spatial and temporal domains. Nevertheless, they depend on explicit low-level visual prompts (e.g., points or boxes) and cannot natively interpret complex conversational text instructions. Conversely, as illustrated in Figure 2 , recent segmentation MLLMs have attempted to bridge language understanding and mask generation, but they remain structurally fragmented. Image segmentation MLLMs (e.g., LISA [ 19 ] ) process textual instructions but are restricted to static images and usually lack visual prompting support. Video segmentation MLLMs (e.g., VISA [ 62 ] , VideoLISA [ 6 ] ) support temporal text-to-mask generation but do not provide a unified architecture for both static images and visual prompts. Achieving a single framework that interprets complex multi-modal instructions, including both text and visual prompts, for segmentation across images and videos remains a critical challenge. In this work, we introduce X2SAM, a framework that unifies diverse image and video segmentation tasks and extends the image-centric any-segmentation paradigm toward a unified image-and-video setting. As illustrated in Figure 1 , X2SAM provides a conversational interface for text-driven and visually prompted segmentation across static images and dynamic videos. To realize this capability and overcome limitations of prior paradigms (Figure 2 ), our approach addresses three technical challenges: (1) Comprehensive Prompt Integration : augmenting LLMs to process interleaved textual instructions and visual prompts (V-Prompts) for both image and video inputs. (2) Spatio-Temporal Task Formulation : casting diverse image segmentation paradigms into a shared formulation that can represent video targets over time. (3) Temporal Coherence via Mask Memory : replacing independent frame-by-frame decoding with a Mask Memory module that interacts with the Mask Decoder and stores guided vision features to maintain mask consistency across video sequences. As illustrated in Figure 3 , we develop a unified MLLM architecture that processes global visual representations and fine-grained visual features. Guided by latent condition embeddings from the LLM, the Mask Decoder works with the newly introduced Mask Memory module to generate temporally consistent segmentation masks. Moreover, we expand the visual prompting capabilities of MLLMs by introducing the Video Visual Grounded (V-VGD) segmentation task. This task equips the model to segment any instance object in a video using interactive visual prompts, grounding targets across frames. As shown in Table 1 , we compare X2SAM with existing methods across inputs, outputs, and tasks. X2SAM is the first to natively support seven segmentation tasks, e.g., generic, open-vocabulary, referring, reasoning, grounded conversation generation, object-centric, and visual grounded segmentation, for image and videos. Figure 2 : Comparison of X2SAM with existing methods, including SAM Series models, Image Segmentation MLLMs, and Video Segmentation MLLMs. X2SAM processes both textual and visual prompts in a shared image-video segmentation framework, improving coverage over prior image-only or video-only segmentation MLLMs. Supported by a unified joint training strategy that accelerates learning across multi-modalities, X2SAM undergoes co-training with a diverse range of image and video datasets. Experimental results show that X2SAM achieves strong performance across image and video benchmarks, with particularly consistent gains on video segmentation tasks, establishing a practical baseline for unified pixel-level spatio-temporal understanding. In summary, our contributions are as follows: • We introduce X2SAM, a unified framework that extends the any segmentation paradigm from images to videos. By integrating an MLLM with a Mask Memory module, X2SAM formulates diverse image and video segmentation tasks into a standardized, temporally consistent format. • We propose a new benchmark, Video Visual Grounded (V-VGD) segmentation, which provides interactive visual prompts for MLLMs to ground and segment instance objects consistently across video frames. • We present a unified joint training strategy to co-train X2SAM on both image and video data. Extensive evaluations show that X2SAM supports a broad set of segmentation tasks, remains competitive on image benchmarks, and achieves strong results on video and out-of-domain evaluations. 2 Related Work Multi-modal Large Language Model. Multi-modal learning has witnessed progressive developments alongside the rapid evolution of Large Language Models (LLMs) [ 3 , 51 ] and multi-modal pre-training methods [ 46 , 15 ] . The field has evolved from early models focused on task-specific fusion and feature extraction [ 22 ] , to generalized, instruction-tuned frameworks leveraging visual feature tokenization [ 35 , 32 ] . While current MLLMs demonstrate remarkable effectiveness in global visual understanding tasks such as image captioning [ 60 ] and VQA [ 1 ] , their capability to generate dense, pixel-level outputs for precise spatial and temporal comprehension remains highly limited. This poses a considerable challenge when directly addressing fine-grained tasks across static images and dynamic video sequences. Image Segmentation MLLMs. Foundation models like SAM [ 18 ] and its extensions [ 49 ] have profoundly impacted the segmentation landscape by introducing visual grounding

Read the complete text