Semantic-Driven Scale and Spatial Selection for Efficient Cross-Modal Alignment in Referring Remote Sensing Image Segmentation
Kun Li, Shengxi Gui, Francesco Nex, Michael Ying Yang · 2026
Read Semantic-Driven Scale and Spatial Selection for Efficient Cross-Modal Alignment in Referring Remote Sensing Image Segmentation by Kun Li, Shengxi Gui, Francesco Nex, Michael Ying Yang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30244.txt.
Introduction
Kun Li, Shengxi Gui, Francesco Nex, and Michael Yang segment objects in aerial images from written descriptions. A user names a target in natural language. The system must find and outline that region in remote sensing imagery. Full fine-tuning of foundation models is expensive for this task. It can also distort the broad knowledge those models carry.
Their framework adapts models by updating only a small parameter share. A dual-encoder adapter design drives the cross-modal interaction. The text adapter distills language proxies from word-level embeddings using learnable queries. The visual adapter refines multi-scale features with a dense extractor. A language-guided mechanism then selects the right scale and spatial regions. Only about two percent of backbone parameters change during tuning. The model still reaches state-of-the-art results on two benchmarks. Efficiency and precision both suit complex aerial scenes.
Catalog record
| CREATOR | Kun Li, Shengxi Gui, Francesco Nex, Michael Ying Yang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30244
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Semantic-Driven Scale and Spatial Selection for Efficient Cross-Modal Alignment in Referring Remote Sensing Image Segmentation Kun Li, Shengxi Gui, Francesco Nex, Michael Ying Yang Abstract Referring Remote Sensing Image Segmentation (RRSIS) seeks to localize and segment the target object or region specified by a natural language expression in a remote sensing image, requiring joint understanding of visual and linguistic information. While existing RRSIS models have benefited from large-scale foundation models ( e.g. , CLIP), they predominantly rely on full fine-tuning. These approaches are computationally intensive and may weaken the generalization ability of pre-trained models, as extensive fine-tuning on significantly smaller downstream datasets can distort the well-structured feature representations learned during large-scale pre-training. Although Parameter-Efficient Tuning (PET) offers a potential alternative, existing PET frameworks primarily focus on single-modal optimization, failing to capture the complex cross-modal dependencies required for multimodal reasoning, while simultaneously struggling to bridge the substantial domain gap between natural scenes and aerial imagery. To address these limitations, we propose a novel framework, Semantic-driven Scale and Spatial Selection for Efficient Cross-modal Alignment (S 4 ECA), which enables effective and efficient cross-modal interaction through parameter-efficient adaptation. Specifically, we design a dual-encoder adapter architecture. The textual adapter employs learnable queries to distill highly semantic language proxies from word-level embeddings, facilitating early grounding. Simultaneously, the visual adapter refines hierarchical feature representations through a multi-scale dense extractor, followed by a language-guided scale and spatial selection mechanism that dynamically emphasizes relevant visual contexts, ensuring precise cross-modal alignment. By updating only 2.4% of the backbone parameters, our proposed model achieves state-of-the-art performance on the RRSIS-D and RefSegRS datasets, demonstrating superior efficiency and precision in complex aerial scenarios. 1 Introduction Referring Remote Sensing Image Segmentation (RRSIS) aims to identify and segment the target region within an aerial image based on a natural language expression. In contrast to traditional vision-only segmentation tasks with fixed semantic categories, RRSIS requires fine-grained cross-modal reasoning to associate open-vocabulary textual expressions with the corresponding visual regions. Beyond its theoretical significance, RRSIS has significant potential for advancing human-centered geospatial intelligence by enabling intuitive language-based interaction with remote sensing imagery. Such capability is valuable for a wide range of applications, including urban monitoring [ 9 , 24 ] , disaster response [ 45 , 18 ] , precision agriculture [ 49 ] , and land-use change survey [ 31 ] , where users often need to identify specific targets through natural language descriptions rather than predefined semantic categories. However, accurate pixel-level localization remains highly challenging due to the substantial variations in object scale, complex spatial relationships among targets, and the cluttered backgrounds in high-resolution aerial imagery. Benefiting from recent advances in vision-language learning [ 44 , 54 , 35 , 26 ] , RRSIS has witnessed rapid progress in multimodal representation learning and cross-modal interaction. Existing methods, such as LGCE [ 58 ] and RMSIN [ 34 ] , generally follow a standard representation-fusion-segmentation paradigm, where hierarchical visual features are progressively integrated with linguistic representations extracted by pre-trained language encoders to facilitate target localization and segmentation. Despite their promising performance, existing RRSIS frameworks primarily rely on full-parameter fine-tuning for domain adaptation (as shown in Figure 1(a) ). Such a paradigm faces two fundamental challenges. First, optimizing all model parameters on remote sensing datasets that are substantially smaller than the original pre-training corpus may compromise the transferability of pre-trained representations, limiting the preservation of general knowledge. Second, updating the entire parameter space introduces considerable computational and memory overhead, reducing scalability and limiting practical deployment in real-world applications. (a) Full fine-tuning. (b) Bridger-based PET. (c) Dual-adapter-based PET. (d) Our S4 adapter-based PET. Figure 1 : Illustration of distinct paradigms for the referring image segmentation task. (a) Traditional full fine-tuning methods update the entire pre-trained vision-language model on the target task. (b) Standard adapter-based parameter-efficient tuning (PET) methods introduce lightweight bottleneck modules (Bridger) into the frozen backbones to reduce parameter overhead. (c) Dual-adapter-based PET methods employ both visual and textual adapters, facilitating the efficient transfer via modular updates. (d) Our proposed S 4 ECA method builds upon the dual-adapter architecture with our novel semantic-driven scale and spatial selection (S4) strategy by updating only a small set of tunable parameters. To address these challenges, Parameter-Efficient Tuning (PET) has emerged as a promising alternative for adapting large-scale foundation models to downstream tasks [ 13 , 3 ] . Instead of updating the entire model, PET freezes most pre-trained parameters and introduces a small set of task-specific trainable components, enabling efficient adaptation while largely preserving the rich knowledge acquired during pre-training. As a result, PET substantially reduces computational and memory costs without sacrificing the representational capacity of pre-trained models. However, existing PET techniques are predominantly applied to single-modality tasks ( e.g. , image classification) and remain inadequately explored in the context of RRSIS. Although a few pioneering studies have extended PET to dense prediction tasks, they primarily focus on intra-modal adaptation, while overlooking the complex cross-modal reasoning required to accurately align free-form textual descriptions with fine-grained visual content. Consequently, they struggle to capture the complementary dependencies between language and vision that are essential for precise pixel-level localization. Moreover, these methods are typically developed on natural image domains and therefore exhibit limited adaptability to remote sensing scenarios, where large-scale spatial variation, dense object distribution, and highly cluttered backgrounds introduce significantly greater complexity. For instance, existing bridger-based designs such as ETRIS [ 53 ] do not explicitly model multi-scale visual structures, which are crucial for handling objects that vary dramatically in size within aerial imagery. In addition, current methods place limited emphasis on textual adaptation, despite the fact that reliable segmentation in complex remote sensing scenes requires precise linguistic grounding to disambiguate visually similar targets. For instance, DETRIS [ 15 ] relies solely on intra-modal updates within its textual adapter design, without explicitly modeling the dynamic interaction between textual representations and evolving visual features. Consequently, these limitations significantly restrict their capacity to achieve robust cross-modal alignment and precise segmentation under the complex and diverse conditions of aerial imagery. In this paper, we propose the S emantic-driven S cale and S patial S election based E fficient C ross-modal A lignment network (S 4 ECA), an effective and efficient RRSIS framework designed for domain-specific adaptation. The framework adopts a dual-adapter architecture that performs modality-specific adaptation for both visual and textual encoders while preserving the majority of pre-trained parameters. Different from existing PET methods that apply static or intra-modal layer-wise tuning (as shown in Figure 1 ), our proposed textual adapter introduces a set of learnable queries to represent high-level semantic language proxies for early grounding. Specifically, a lightweight language enhancer is first employed to capture local and global dependencies among textual tokens, after which the learnable queries aggregate complementary semantic information from the linguistic features. These semantic proxies are subsequently interacted with visual representations through a lightweight cross-attention mechanism, enabling early cross-modal interaction. By jointly modeling local textual structures and global semantic cues, the proposed design enhances language understanding while achieving efficient textual adaptation with substantially lower computational cost than conventional dense cross-attention. To effectively adapt visual representations to the complex characteristics of remote sensing imagery, a dedicated convolutional module is further designed to extract multi-scale visual features. Nevertheless, visual features alone often contain