X-SAM: From Segment Anything to Any Segmentation

Hao Wang, Limeng Qiao, Zequn Jie, Zhijian Huang, Chengjian Feng, Qingfang Zheng, Lin Ma, Xiangyuan Lan, Xiaodan Liang · 2025

Read X-SAM: From Segment Anything to Any Segmentation by Hao Wang, Limeng Qiao, Zequn Jie, Zhijian Huang, Chengjian Feng, Qingfang Zheng, Lin Ma, Xiangyuan Lan, Xiaodan Liang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-04655.txt.

Introduction

Hao Wang and co-authors present X-SAM, a model that widens image segmentation from segmenting anything to segmenting in any way, in a 2025 paper. Big language models know a lot but see pixels poorly. The Segment Anything Model cuts out objects from visual prompts, yet it juggles multiple masks and category tasks awkwardly and cannot unify all segmentation jobs.

X-SAM rebuilds the idea as a multimodal language model framework. It adds a new task, visually grounded segmentation, where interactive visual prompts mark every instance to cut out, giving the model pixel-level reading of a scene. A shared training plan lets many datasets teach together. Tests show state-of-the-art results across a wide range of segmentation benchmarks.

Catalog record

CREATORHao Wang, Limeng Qiao, Zequn Jie, Zhijian Huang, Chengjian Feng, Qingfang Zheng, Lin Ma, Xiangyuan Lan, Xiaodan Liang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-04655

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

X-SAM: From Segment Anything to Any Segmentation Hao Wang, Limeng Qiao, Zequn Jie, Zhijian Huang, Chengjian Feng, Qingfang Zheng, Lin Ma, Xiangyuan Lan, Xiaodan Liang Abstract Large Language Models (LLMs) demonstrate strong capabilities in broad knowledge representation, yet they are inherently deficient in pixel-level perceptual understanding. Although the Segment Anything Model (SAM) represents a significant advancement in visual-prompt-driven image segmentation, it exhibits notable limitations in multi-mask prediction and category-specific segmentation tasks, and it cannot integrate all segmentation tasks within a unified model architecture. To address these limitations, we present X-SAM, a streamlined Multimodal Large Language Model (MLLM) framework that extends the segmentation paradigm from segment anything to any segmentation . Specifically, we introduce a novel unified framework that enables more advanced pixel-level perceptual comprehension for MLLMs. Furthermore, we propose a new segmentation task, termed Visual GrounDed (VGD) segmentation, which segments all instance objects with interactive visual prompts and empowers MLLMs with visual grounded, pixel-wise interpretative capabilities. To enable effective training on diverse data sources, we present a unified training strategy that supports co-training across multiple datasets. Experimental results demonstrate that X-SAM achieves state-of-the-art performance on a wide range of image segmentation benchmarks, highlighting its efficiency for multimodal, pixel-level visual understanding. Code is available at https://github.com/wanghao9610/X-SAM . 1 Introduction Multi-modal Large Language Models (MLLMs) have exhibited substantial advancements alongside the rapid development of Large Language Models (LLMs) ( Bai et al. 2023 ; Touvron et al. 2023a ; Touvron et al. 2023b ; Abdin et al. 2024 ) and multi-modal pre-training methods ( Radford et al. 2021 ; Jia et al. 2021 ; Zhai et al. 2023 ) . These models have shown remarkable effectiveness in a wide range of applications, including image captioning ( Xu et al. 2015 ) , VQA ( Antol et al. 2015 ) , and visual editing ( Chen et al. 2018 ) . Nevertheless, a significant impediment to developing a truly generalized model remains: contemporary MLLMs are restricted to generating solely textual outputs. This limitation poses a considerable challenge in directly addressing tasks that require pixel-level comprehension of visual data, such as image segmentation, which is the most critical task in the field of computer vision. The Segment Anything Model (SAM) represents a foundational segmentation model that demonstrates exceptional efficacy in generating dense segmentation masks and has inspired the development of various segmentation tasks, such as high-quality segmentation ( Ke et al. 2023 ) , matching anything ( Li et al. 2024c ) , and tracking anything ( Rajič et al. 2025 ) . Nevertheless, SAM’s architecture is fundamentally constrained by its dependency on visual prompts, which significantly limits its direct applicability to a wide range of image segmentation tasks, including generic (semantic, instance, panoptic) segmentation, referring segmentation, and open-vocabulary (OV) segmentation, among others. Achieving a unified framework capable of addressing various image segmentation tasks remains a challenging problem. Fig. 1: Illustration of the capabilities of X-SAM. (a). Text query tasks: Generic(Gen.), Referring(Ref.), Reasoning(Rea.), and Grounded Conversation Generation(GCG) segmentation, etc. . (b). Vision query tasks: Interactive(Inter.) and Visual GrounDed(VGD) segmentation for single and cross-image. (c). X-SAM outperforms existing MLLMs on all segmentation benchmarks. Best viewed in zoom. In this work, we introduce X-SAM, an innovative framework that unifies diverse image segmentation tasks, expanding the segmentation paradigm from segment anything to any segmentation . To accomplish this objective, our approach addresses three critical technical challenges: (1) Task formulation : Transforming SAM into a versatile segmentation architecture with cross-task applicability. (2) Modality enhancement : Augmenting LLMs with multimodal input processing capabilities. (3) Unified framework : Developing a cohesive approach to effectively facilitate comprehensive segmentation applications across diverse domains. First, we develop a unified segmentation MLLM architecture that incorporates a unified mask decoder capable of generating segmentation masks suitable for generalized image segmentation tasks. Second, we expand the multimodal capabilities of MLLMs to process not only textual queries but also visual queries. Specifically, we introduce a novel task termed Visual Grounded (VGD) segmentation, which segments all instance objects with interactive visual prompts in an image. This task introduces visual guide modalities into large language models (LLMs). Furthermore, we propose a unified input format and training methodology that reformulates segmentation tasks within a unified framework, thus optimizing the adaptation of MLLMs to diverse image segmentation tasks. As shown in Fig. 1 and Tab. 1 , we present the comprehensive capabilities of X-SAM and compare them with those of other methods. Our proposed framework exhibits capabilities in processing text query-based tasks, such as generic segmentation and referring segmentation, while simultaneously accommodating vision query-based tasks such as interactive segmentation ( Zhang et al. 2024d ) and our novel VGD segmentation, which functions effectively in both single-image and cross-image contexts. Furthermore, X-SAM leverages the reasoning and generative capacities of LLMs, thereby enabling advanced reasoning segmentation and Grounded Conversation Generation (GCG) ( Rasheed et al. 2024 ) segmentation. X-SAM undergoes co-training with a diverse range of datasets. We perform a comprehensive evaluation on more than twenty segmentation datasets across seven distinct image segmentation tasks, even including the image conversion task. X-SAM achieves the state-of-the-art performance across all image segmentation benchmarks, and establishes a robust new baseline for unified pixel-level image understanding. In summary, our contributions are as follows: • We introduce X-SAM, a novel unified framework that extends the segmentation paradigm from segment anything to any segmentation . Our approach formulates diverse image segmentation tasks into a standardized segmentation format. • We propose a new image segmentation benchmark, Visual GrounDed (VGD) Segmentation, which provides visual grounded prompts for MLLMs to segment instance objects in images. The benchmark introduces user-friendly inputs to ground the segmentation objects and guide the MLLMs to output the segmentation masks. • We present a unified multi-stage training strategy to co-train X-SAM with a diverse range of datasets, and conduct extensive evaluations on more than twenty image segmentation benchmarks, achieving state-of-the-art performance on all of them. This establishes a new strong baseline for unified pixel-level perceptual understanding in MLLMs. 2 Related Work Multi-modal Large Language Model. Multi-modal learning has evolved from early models focused on task-specific fusion and feature extraction ( Li et al. 2022b ) , to leveraging large language models ( Brown et al. 2020 ; Touvron et al. 2023a ; Touvron et al. 2023b ) for generalized, instruction-tuned multi-task benchmarks ( Li et al. 2023a ; Liu et al. 2024c ; Hudson et al. 2019 ) . LLaVA ( Liu et al. 2024a ; Liu et al. 2024b ; Liu et al. 2023a ) introduced visual feature tokenization, inspiring advances in visual representation ( Yuan et al. 2024b ) , specialized vision extensions ( Lai et al. 2024 ; Lin et al. 2023 ; Dong et al. 2024a ; Dong et al. 2024b ; Zhang et al. 2023 ; Ren et al. 2024 ; Zhang et al. 2024a ; Zang et al. 2025 ) , and language-guided segmentation ( Li et al. 2024e ; Zhang et al. 2024b ) . However, most progress remains task-specific. To our knowledge, we are the first to successfully implement a comprehensive approach, opening new directions for image segmentation. Tab. 1: Comparison of Capability. We compare different methods on both segmentation-specific ( Gray ) and MLLM-based. Method Text Query Vision Query Gen.

Read the complete text