Controllable-LPMoE: Adapting to Challenging Object Segmentation via Dynamic Local Priors from Mixture-of-Experts

Yanguang Sun, Jiawei Lian, Jian Yang, Lei Luo · 2025

Read Controllable-LPMoE: Adapting to Challenging Object Segmentation via Dynamic Local Priors from Mixture-of-Experts by Yanguang Sun, Jiawei Lian, Jian Yang, Lei Luo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21114.txt.

Introduction

Yanguang Sun leads three co-authors on Controllable-LPMoE, a fine tuning method that adapts large foundation models to hard object segmentation using dynamic local priors from a mixture of experts. Full fine tuning updates enormous parameter counts and creates a training efficiency bottleneck.

Prompt based tuning of frozen models is cheaper but prompts lack built in semantic priors, limiting adaptability. Their paradigm supplies dynamic local priors while keeping trainable parameters few. Challenging segmentation targets, where boundaries are subtle and classes shift, stand to gain most from priors that adjust to each image region.

Catalog record

CREATORYanguang Sun, Jiawei Lian, Jian Yang, Lei Luo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-21114

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Controllable-LPMoE: Adapting to Challenging Object Segmentation via Dynamic Local Priors from Mixture-of-Experts Yanguang Sun, Jiawei Lian, Jian Yang, Lei Luo Abstract Large-scale foundation models provide powerful feature representations for downstream object segmentation tasks. However, when adapted to specific tasks through the full-parameter fine-tuning, the enormous parameters being updated often results in significant computational overhead, creating a bottleneck in training efficiency. Although existing methods attempt to fine-tune frozen models by directly embedding trainable prompts, these prompts lack inherent semantic priors, limiting the adaptability of large-scale models. In this paper, we propose a novel dynamic priors-based fine-tuning paradigm with fewer trainable parameters, dubbed Controllable-LPMoE, which adaptively modulates frozen foundation models by dynamically controlling local priors to enhance fine-grained perception for specific segmentation tasks. More specifically, we construct a lightweight dynamic mixed local priors extractor that captures diverse local priors from input images through heterogeneous convolutions while employing a gating network to dynamically output expert priors required for the subsequent fine-tuning. Furthermore, we design a bi-directional interaction adapter that employs cosine-aligned deformable attention and channel-oriented adaptive scale enhancement to interact and restructure between frozen and trainable features, achieving efficient fine-tuning. Extensive experiments validate the superiority of our Controllable-LPMoE approach, demonstrating excellent segmentation performance compared to 31 state-of-the-art (SOTA) methods and adaptability to multiple binary object segmentation tasks. 1 Introduction Binary object segmentation, as a fundamental task in computer vision, has been widely studied and encompasses multiple directions, including camouflaged object detection (COD) [ 22 , 59 , 23 , 52 , 58 ] , salient object detection (SOD) [ 19 , 43 , 69 , 89 ] , polyp segmentation (PS) [ 85 , 16 , 68 ] , skin lesion segmentation (SLS) [ 20 , 76 ] , shadow detection (SD) [ 42 , 44 , 34 ] , glass detection (GD) [ 17 , 41 , 24 ] , among others. Over the past few years, numerous deep learning-based methods [ 25 , 19 , 30 , 58 , 70 , 61 ] have been proposed, contributing to substantial advances in this field. Figure 1 : Training paradigms in binary object segmentation tasks: (a) Full-parameter fine-tuning, which updates all model parameters for task adaptation; (b) Prompt-based fine-tuning, which guides learning through trainable prompt embeddings; (c) The proposed dynamic priors-based fine-tuning, which utilizes dynamically controllable local priors to efficiently adapt large-scale models [ 1 , 88 ] . Additionally, we provide the trainable parameters using different paradigm methods and their IoU accuracy on the challenging COD10K [ 15 ] dataset. The efficient extraction and encoding of high-quality features from input images is a critical factor for achieving accurate binary object segmentation tasks. Early research [ 51 , 69 , 84 , 56 ] usually utilizes pre-trained convolutional neural networks ( e.g. , ResNet50 [ 26 ] with 25.6M parameters or VGG16 [ 54 ] with 14.7M parameters) as feature encoders, which have relatively few parameters and can be adapted to specific tasks through full-parameter fine-tuning. Later, Vision Transformers [ 12 , 45 , 67 ] exhibit strong feature modeling by integrating self-attention and feed-forward networks at each encoding layer. Based on these structures, employing Transformers [ 30 , 43 , 46 , 52 , 58 ] via full-parameter fine-tuning (as depicted in Fig. 1 (a)) has become the mainstream architecture in binary object segmentation tasks, achieving a significant performance breakthrough. However, during the training process, a series of issues appeared consecutively, the most prominent being a sharp increase in memory consumption and a notable decline in training speed, both resulting from the substantial increase in parameters within Vision Transformers [ 12 , 67 ] . For example, in the challenging COD task, the ZoomXNet [ 52 ] approach adopts PVTv2-b5 [ 67 ] , which has 82M parameters. The trainable parameters for the FSPNet [ 30 ] , FSEL [ 58 ] , and CamoFormer [ 79 ] methods are 273.7M, 67.1M, and 71.3M. Similarly, in the VST++ [ 43 ] model for the SOD task, 112.2M parameters need to be updated, etc . It is evident that when larger-scale Transformer models ( e.g. , BEiT-L [ 1 ] with 307M parameters or UniPerceiver-L [ 88 ] with 302M parameters) with deeper layers and more parameters, which possess stronger modeling capabilities, are used for feature encoding, the feasibility of this fine-tuning paradigm becomes negligible. Recently, the prompt-based fine-tuning paradigm [ 32 ] has been proposed in visual recognition tasks, which embeds trainable prompts (as shown in Fig. 1 (b)) with few parameters into frozen Transformer layers to enable large-scale models to adapt to specific tasks. Inspired by this, EVP [ 44 ] and VSCode [ 46 ] introduce prompt learning into the binary object segmentation task, acquiring task-specific knowledge through different prompts to fine-tune Transformer [ 45 , 75 ] to segment objects in various scenarios. Although the training efficiency of these models [ 44 , 46 ] has improved considerably, the segmentation accuracy remains unsatisfactory. As shown in Fig. 1 , on the extremely difficult COD10K [ 15 ] dataset, the performance of EVP [ 44 ] and VSCode [ 46 ] is considerably lower than that of ZoomXNet [ 52 ] and FSEL [ 58 ] , which adopt full-parameter fine-tuning. The reasons for this can be attributed to two aspects: 1) it fails to fully leverage the powerful modeling of large-scale models, and 2) it is closely related to the inherent properties of prompts. In particular, direct-generated prompts lack prior knowledge, making it challenging to refine object details during iterative training. Furthermore, simply embedding trainable prompts fails to adequately incorporate task-specific knowledge into frozen features, influencing the final segmentation performance. Taking these reasons into account, we propose a novel dynamic priors-based fine-tuning paradigm in this paper, named Controllable-LPMoE. As illustrated in Fig. 1 , our method involves few trainable parameters ( i.e. , 23.4M), yet achieves high segmentation accuracy, benefiting from the efficient fine-tuning of large-scale models. Technically, we construct a lightweight dynamic mixed local priors (DMLP) extractor to generate dynamically controllable local priors with task-specific knowledge through multiple heterogeneous convolutions [ 4 , 11 , 18 , 28 ] and a mixture-of-experts (MoE) strategy [ 35 ] from input images for subsequent fine-tuning of large-scale foundation models. Moreover, we design a bi-directional interaction (BDI) adapter to facilitate information transfer between the trainable and frozen features, progressively reconstructing their internal information through iterative updates of cosine-aligned deformable attention and channel-oriented adaptive scale enhancement components. Ultimately, optimized features not only retain powerful universal representations from large-scale models but also incorporate task-specific knowledge, making them efficiently adaptable to segmentation tasks. Extensive experiments on 18 widely-used benchmark datasets from 6 binary object segmentation tasks demonstrate that our Controllable-LPMoE model consistently outperforms 31 state-of-the-art (SOTA) methods. In summary, the main contributions can be summarized as follows: \bullet A novel dynamic priors-based fine-tuning paradigm is proposed for adapting large-scale models to binary object segmentation tasks through fewer trainable parameters. \bullet A lightweight dynamic mixed local priors (DMLP) extractor is designed to dynamically capture various local priors using different convolutions and the MoE strategy. \bullet An efficient bi-directional interaction (BDI) adapter is introduced to reconstruct the representations of trainable and frozen features, leveraging them through

Read the complete text