InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models

Cong Wei, Yujie Zhong †, Haoxian Tan, Yingsen Zeng, Yong Liu, Zheng Zhao, Yujiu Yang † · 2024

Read InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models by Cong Wei, Yujie Zhong †, Haoxian Tan, Yingsen Zeng, Yong Liu, Zheng Zhao, Yujiu Yang † (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-14006.txt.

Introduction

Cong Wei and six co-authors present a 2024 paper on one system that cuts out objects in both photos and videos from text orders. Past systems were built apart for stills and for clips, though the tasks are much alike. The authors name the joined task Instructed Visual Segmentation and build one pipeline for it.

Their system pairs a large vision-language model with a video reader that tracks objects across frames. A fusion step blends broad and fine text cues under visual guidance. Trained end to end on many tasks, the system beat prior tools on a range of photo and video cutting tests.

Catalog record

CREATORCong Wei, Yujie Zhong †, Haoxian Tan, Yingsen Zeng, Yong Liu, Zheng Zhao, Yujiu Yang †
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-14006

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models Cong Wei, Yujie Zhong †, Haoxian Tan, Yingsen Zeng, Yong Liu, Zheng Zhao, Yujiu Yang † Abstract Boosted by Multi-modal Large Language Models (MLLMs), text-guided universal segmentation models for the image and video domains have made rapid progress recently. However, these methods are often developed separately for specific domains, overlooking the similarities in task settings and solutions across these two areas. In this paper, we define the union of referring segmentation and reasoning segmentation at both the image and video levels as Instructed Visual Segmentation (IVS). Correspondingly, we propose InstructSeg, an end-to-end segmentation pipeline equipped with MLLMs for IVS. Specifically, we employ an object-aware video perceiver to extract temporal and object information from reference frames, facilitating comprehensive video understanding. Additionally, we introduce vision-guided multi-granularity text fusion to better integrate global and detailed text information with fine-grained visual guidance. By leveraging multi-task and end-to-end training, InstructSeg demonstrates superior performance across diverse image and video segmentation tasks, surpassing both segmentation specialists and MLLM-based methods with a single model. Our code is available here . Figure 1 : We define Instructed Visual Segmentation (IVS) as the union of four text-guided segmentation tasks across image and video domains: referring expression segmentation (RES), reasoning segmentation (ReasonSeg), referring video object segmentation (R-VOS) and reasoning video object segmentation (ReasonVOS). InstructSeg can handle all the IVS tasks in one model with excellent performance. † † footnotetext: † Corresponding authors. Table 1 : The comparison of capabilities of different methods. Our InstructSeg can tackles more comprehensive Instructed Visual Segmentation tasks with a simplified end-to-end framework and achieves better performance. Method End-to-End Instructed Visual Segmentation RES ReasonSeg R-VOS ReasonVOS Segmentation Specialists VLT [ 11 ] , LAVT [ 48 ] , ReLA [ 24 ] , PolyFormer [ 26 ] ✓ ✓ ReferFormer [ 44 ] , OnlineRefer [ 43 ] , SOC [ 29 ] ✓ ✓ MLLM-based Generalists LISA [ 18 ] , PixelLM [ 35 ] , LaSagnA [ 41 ] ✓ ✓ ✓ GSVA [ 45 ] , GLaMM [ 34 ] , OMG-LLaVA [ 51 ] , PSALM [ 53 ] ✓ ✓ VISA [ 47 ] ✓ ✓ ✓ ✓ InstructSeg (Ours) ✓ ✓ ✓ ✓ ✓ 1 Introduction In recent years, language-instructed segmentation tasks, such as referring expression segmentation (RES) [ 11 , 40 , 48 , 26 ] and referring video object segmentation (R-VOS) [ 13 , 44 , 43 ] , have attracted significant attention in the field of computer vision. These tasks require models to segment objects in images or videos based on natural language descriptions, offering greater potential for practical applications but also presenting a more substantial challenge compared to traditional class-based segmentation tasks like semantic segmentation, which depend on predefined categories. Multi-modal Large Language Models (MLLMs) are distinguished by their ability to handle versatile inputs and their impressive performance in visual-language reasoning, making them a preferred choice for these language-instructed segmentation tasks in both image and video domains. Despite the similarity in task settings and solutions, current research often overlooks the collaborative effect of these tasks in the context of multi-task optimization, treating them separately. Recent works in the image domain, such as LISA [ 18 ] , PixelLM [ 35 ] , and PSALM [ 53 ] , have demonstrated promising results in universal image segmentation by leveraging MLLMs’ embedded reasoning capabilities and inherent world knowledge. However, models designed for static images typically fail to effectively capture the temporal coherence within videos , either by naively aggregating video frames or by processing video frames individually, thereby incurring excessive computational costs. Meanwhile, recent work in the video domain, VISA [ 47 ] , has sought to address these challenges by leveraging the combination of multiple specialists for text-guided image and video segmentation. Nonetheless, VISA requires the integration of each specialist, necessitating the use of additional pre-trained Video-LLMs and a video object tracker. The complex model and cumbersome pipeline impede the potential for widespread application and end-to-end collaborative optimization in both image and video domains. In this work, we unify referring segmentation and reasoning segmentation within both the image and video domains under the framework of Instructed Visual Segmentation (IVS) . IVS requires models to segment targets from visual inputs based on detailed textual instructions. As shown in Tab. 1 , IVS encompasses traditional referring segmentation tasks, such as referring expression segmentation (RES) and referring video object segmentation (R-VOS), as well as more complex reasoning-based segmentation tasks, including reasoning segmentation (ReasonSeg) and reasoning video object segmentation (ReasonVOS). Accordingly, we introduce an end-to-end pipeline, InstructSeg, along with two meticulous designs: Object-aware Video Perceiver and Vision-guided Multi-granularity Text Fusion. This pipeline enables the implementation of all tasks within IVS using a single MLLM and segmentation decoder. Specifically, Object-aware Video Perceiver is designed to extract both the temporal and object information from reference frames guided by language instructions referring to or reasoning about the objects of interest. Vision-guided Multi-granularity Text Fusion is proposed to comprehend long text instructions and complex scenarios like “A bride and groom often walk together down the aisle during a wedding ceremony. What object in the picture is the bride most likely holding during this moment?”. Instead of averaging or summarizing the embeddings of multiple text tokens, we integrate both global and detailed information of text instructions into the multi-granularity text embeddings, facilitating better comprehension of the intentions behind the text instructions. Extensive experiments on various Instructed Visual Segmentation benchmarks demonstrate the powerful reasoning and segmentation abilities of our InstructSeg. Our contributions are as follows: • We integrate referring segmentation and reasoning segmentation across both image and video domains within the framework of Instructed Visual Segmentation (IVS). Introducing InstructSeg, we offer an end-to-end unified pipeline for all IVS tasks, leveraging Multi-modal Large Language Models for language-instructed pixel-level reasoning and classification. This model effectively handles tasks across both image and video domains while maintaining a high level of simplicity. • We propose Object-aware Video Perceiver and Vision-guided Multi-granularity Text Fusion modules to fully exploit temporal and object information, enhancing the understanding of both global and detailed text instructions. • InstructSeg achieves state-of-the-art results on diverse Instructed Visual Segmentation benchmarks across both image and video domains, demonstrating the effectiveness of our simplified pipeline. Additionally, InstructSeg delivers competitive performance on multiple Multi-modal benchmarks. 2 Related Work Referring segmentation in image and video. The referring expression segmentation (RES) task aims at labeling the pixels of an image that represent an object instance referred to by a linguistic expression. CRIS [ 40 ] , CGFormer [ 39 ] , CoupAlign [ 52 ] firstly use visual language decoder to propagate text information to pixel-level visual features, and then bring relevant features closer based of contrastive Learning. MCN [ 28 ] , RefTR [ 22 ] , and X-Decoder [ 56 ] enhance the fusion of textual and visual features and utilize multi-task collaborative learning to strengthen the segmentation performance. The referring video object segmentation (R-VOS) extends the RES task to the domain of videos, where the goal is to track and segment the object corresponding to a given natural language description in a video. The method [ 4 ] extracts frames from the

Read the complete text