MPerS: Dynamic MLLM MixExperts Perception-Guided Remote Sensing Scene Segmentation

Ziyi Wang, Xianping Ma, Ziyao Wang, Hongyang Zhang, Man On Pun · 2026

Read MPerS: Dynamic MLLM MixExperts Perception-Guided Remote Sensing Scene Segmentation by Ziyi Wang, Xianping Ma, Ziyao Wang, Hongyang Zhang, Man On Pun (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10769.txt.

Introduction

Wang and four colleagues propose MPerS, a system for segmenting remote-sensing scenes that lets multimodal language models perceive them from several expert viewpoints. Prompts generate high-quality captions, DINOv3 extracts dense visual features of land cover, and a Dynamic MixExperts module blends the most useful text semantics. Linguistic Query Guided Attention then steers visual features, giving superior results on three public segmentation datasets.

Catalog record

CREATORZiyi Wang, Xianping Ma, Ziyao Wang, Hongyang Zhang, Man On Pun
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34551
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10769

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MPerS: Dynamic MLLM MixExperts Perception-Guided Remote Sensing Scene Segmentation Ziyi Wang, Xianping Ma, Ziyao Wang, Hongyang Zhang, Man On Pun Abstract The multimodal fusion of images and scene captions has been extensively explored and applied in various fields. However, when dealing with complex remote sensing (RS) scenes, existing studies have predominantly concentrated on architectural optimizations for integrating textual semantic information with visual features, while largely neglecting the generation of high-quality RS captions and the investigation of their effectiveness in multimodal semantic fusion. In this context, we propose the Dynamic MLLM Mixture-of-Experts Perception-Guided Remote Sensing Scene Segmentation, referred to as MPerS. We design multiple prompts for MLLMs to generate high-quality RS captions, enabling MLLMs to perceive RS scenes from diverse expert perspectives. DINOv3 is employed to extract dense visual representations of land-covers. We design a Dynamic MixExperts module that adaptively integrates the most effective textual semantics. Linguistic Query Guided Attention is constructed to utilize textual semantic information to guide visual features for precise segmentation. The MLLMs include LLaVA, ChatGPT, and Qwen. Our method achieves superior performance on three public semantic segmentation RS datasets. 1 Introduction Semantic segmentation 12 is a fundamental task in computer vision. In remote sensing scenes (RS-Scence) with high resolution and extensive spatial coverage, it serves as a crucial technique supporting various remote sensing (RS) interpretation applications. Semantic segmentation in RS scenes assigns each pixel to a specific semantic category. In traditional research, semantic segmentation in RS-Scence has mainly relied on single-modality high-resolution imagery 22 ; 32 . As a result, most existing studies have concentrated on effectively extracting land-cover features to facilitate segmentation in downstream tasks. However, due to the complexity of RS-Scenes and the dense distribution of land-covers, single-modality image data requires a large number of precise manual annotations 40 , which are time-consuming and costly. Furthermore, single-modality image data provide a very limited perception of RS-Scence, thereby imposing substantial constraints on the performance of models trained exclusively on such data. From the perspective of human scene perception, we likewise integrate multiple heterogeneous sensory modalities to comprehend and analyze the inherent characteristics and constituent elements of the current scene. Consequently, accurate perception and fusion of multimodal data better represent real-world scenes and are highly applicable to practical RS interpretation. The diversity of sensors and the expanding volume of RS imagery have also increasingly driven interest in multimodal approaches for RS semantic segmentation. Perception in RS-Scence has expanded from single-image data sources to multispectral imagery 1 , synthetic aperture radar data 19 , and digital surface model 35 , enabling scene understanding from multiple perspectives. Researchers have developed various fusion strategies to leverage information from these data sources. In recent years, visual-language representation learning 18 ; 52 has emerged as an important research area in computer vision. Its goal is to leverage deep learning techniques to extract unified cross-modal feature representations from image-text pairs, thereby enhancing various multimodal tasks. Advances in multimodal large language models (MLLMs) have further propelled the field, enabling their application to diverse downstream tasks 51 . For dense RS-Scenes, the fusion of textual semantic features with visual features is undoubtedly important. The way textual captions are generated for perceiving scenes, as well as their effectiveness, directly impacts the performance of segmentation after visual-textual fusion. To address the aforementioned issues, we propose MPerS, a DINOv3-based semantic segmentation method, where the segmentation is guided by captions generated by diverse MLLM experts, each of whom perceives remote sensing scene characteristics at varying levels of domain expertise. Previous methods 36 rely solely on simple descriptive or category texts to perceive RS scenes. Our work aims to generate high-quality captions that provide textual semantic information conducive to downstream segmentation tasks and incorporates Guided Attention fusion between images and text features. The workflow of our work is show in Fig. . The main contributions are as follows: • We propose MPerS. To our knowledge, this is the first work to investigate the effectiveness of text-guided captions generated by different MLLMs in aiding multimodal dense RS-Scene segmentation. • Three types of MLLMs are leveraged to build a Dynamic MixExperts text encoder, designed to extract the textual semantic information that most effectively guides image features for downstream segmentation tasks. • Linguistic Query Guided Attention is designed to fully utilize textual information, ensuring that textual semantics guide image features for more accurate segmentation. Our method demonstrates excellent performance on three public RS datasets: Potsdam, Vaihingen, and SynDrone. 2 Related Work 2.1 Semantic Segmentation for Remote Sensing Scenes With the increasing application of deep learning-based semantic segmentation in remote sensing Sences (RS-Scence), many approaches have achieved remarkable performance. For instance, multi-scale Convolutional Neural Networks (CNNs) have been designed to efficiently extract land-cover features 14 ; 39 , while various attention mechanisms have been tailored to the unique characteristics of RS-Sences 16 ; 17 . Additionally, several studies have proposed hybrid architectures that combine Transformers and CNNs 38 ; 34 ; 5 ; 3 ; 46 ; 21 , aiming to achieve fine-grained land-cover segmentation. The aforementioned methods have achieved excellent segmentation performance. However, obtaining pixel-level annotations for RSIs is time-consuming and labor-intensive, and the segmentation performance remains significantly constrained. Accordingly, multimodal methods 24 ; 25 have garnered increasing attention and enhance the perception of RS scenes. 2.2 Remote Sensing Scenes Captioning Unlike natural scenes, RS-Scence contain a large number of densely and complexly distributed land-cover objects 13 . RS-Scene captions should not only convey the categories and quantities of land-cover objects, but also precisely capture the spatial relationships and interactions between different land-cover types. By appropriately designing text-assisted tasks, the visual features extracted by the model can be effectively optimized. Zhao et al. 49 proposed a network capable of generating pixel-level segmentation results for both key objects and image descriptions. Ye et al. 44 introduced a multi-label classification task to incorporate prior knowledge, which is then fed into the decoder to improve the accuracy of generated captions. Furthermore, Zhao and Xiong 48 manually annotated object regions in the RS-Scenes Captioning dataset and refined the decoder to enable the model to more effectively utilize fine-grained and regional features. Multimodal Large language model (MLLM) techniques 28 offer significant advantages in the field of natural language processing. Hu et al. 8 proposed a large-scale remote sensing model for RS-Scenes Captioning and RS visual question answering tasks, and also provided a fine-grained RS-Scenes dataset. Yang et al. 43 proposed a two-stage vision-language training method to align visual features with textual information. Compared to traditional caption setting methods, leveraging MLLMs can produce high-quality captions for RS-Scenes. 2.3 Vision-Language Research in Remote Sensing In previous work, learning paradigms for computer vision tasks have predominantly followed pretraining and fine-tuning strategy. Models are initially pretrained on natural images and then fine-tuned on downstream RS tasks, demonstrating excellent performance 4 ; 11 ; 42 . Building on this, an increasing number of researchers have combined multimodal learning with this approach for computer vision tasks 47 . Lu et al. 23 used a co-attention mechanism to capture correlations between modalities and learn joint visual-language representations. Li et al. 15 proposed a learning method that combines visual and language representations, using momentum distillation to pre-align them and improve information fusion from the two modalities.

Read the complete text