Patch-MoE Mamba: A Patch-Ordered Mixture-of-Experts State Space Architecture for Medical Image Segmentation

Diego Adame, Fabian Vazquez, Jose A. Nuñez, Huimin Li, Jinghao Yang, Erik Enriquez, DongChul Kim, Haoteng Tang, Bin Fu, Pengfei Gu · 2026

Read Patch-MoE Mamba: A Patch-Ordered Mixture-of-Experts State Space Architecture for Medical Image Segmentation by Diego Adame, Fabian Vazquez, Jose A. Nuñez, Huimin Li, Jinghao Yang, Erik Enriquez, DongChul Kim, Haoteng Tang, Bin Fu, Pengfei Gu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17719.txt.

Introduction

Diego Adame and nine colleagues propose Patch-MoE Mamba, a design for outlining structures in medical scans. It keeps nearby pixels together while reading at several scales. Standard convolutional nets miss long-range links, and transformers grow costly with image size. So the team scans patches in order and fuses scan directions through expert modules that adapt to object size and shape. Tests on five polyp benchmarks and two skin-lesion sets show the approach works broadly.

Catalog record

CREATORDiego Adame, Fabian Vazquez, Jose A. Nuñez, Huimin Li, Jinghao Yang, Erik Enriquez, DongChul Kim, Haoteng Tang, Bin Fu, Pengfei Gu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24888
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17719

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Patch-MoE Mamba: A Patch-Ordered Mixture-of-Experts State Space Architecture for Medical Image Segmentation Diego Adame, Fabian Vazquez, Jose A. Nuñez, Huimin Li, Jinghao Yang, Erik Enriquez, DongChul Kim, Haoteng Tang, Bin Fu, Pengfei Gu Abstract CNN- and Transformer-based architectures have achieved strong performance in medical image segmentation, but CNNs are limited in modeling long-range dependencies, while Transformers often suffer from quadratic computational and memory complexity. State space models, especially Mamba-based networks, offer an efficient alternative with linear sequence complexity. However, existing Mamba segmentation models still face two limitations: pixel-wise directional scanning can disrupt local 2D spatial structure, and simple summation-based fusion of scan directions cannot adapt well to diverse object sizes, shapes, and boundaries. To address these issues, we propose Patch-MoE Mamba , a patch-ordered mixture-of-experts state space architecture for medical image segmentation. It introduces a hierarchical patch-ordered scanning mechanism that preserves local spatial neighborhoods while capturing multi-scale context, and an MoE-based directional fusion module that adaptively combines multiple Mamba scanner outputs using four directional experts, a learnable concatenation expert, and residual directional aggregation. Experiments on five public polyp segmentation benchmarks and the ISIC 2017/2018 skin lesion segmentation datasets demonstrate the effectiveness and generality of Patch-MoE Mamba. I Introduction Medical image segmentation is a fundamental task in computer-assisted diagnosis and treatment planning [ 1 , 2 , 3 , 4 ] . Advanced convolutional neural networks (CNNs) and Transformer-based architectures have achieved strong performance in this task [ 5 , 6 , 7 , 8 ] . However, CNNs are inherently limited by local receptive fields and thus may struggle to capture long-range dependencies, while Transformers often require large-scale labeled data and incur high computational and memory costs. Recently, state space models, especially Mamba-based architectures, have attracted increasing attention for visual recognition and medical image segmentation because they can model global dependencies with linear complexity in sequence length [ 9 , 10 ] . Despite their promise, existing Mamba-based segmentation networks still face two important limitations. First, most image Mamba models convert a 2D feature map into a 1D sequence and perform pixel-wise scanning along fixed directions [ 9 , 10 , 11 , 12 ] . Although this design enables sequential modeling, it can weaken local 2D spatial coherence: spatially adjacent pixels may become distant in the scan sequence, and consecutive sequence elements may not be close in the image plane. This is undesirable for dense prediction tasks, where accurate boundary localization and local structure preservation are critical. Second, current Mamba-based segmentation methods typically fuse directional scan outputs through simple summation [ 13 , 11 , 12 , 14 ] . Such fixed aggregation treats all directions and receptive fields as equally important at every spatial location. However, medical targets often vary substantially in size, shape, texture, and boundary complexity. Small or low-contrast lesions may require fine-scale local modeling, whereas larger or irregular structures may benefit from broader contextual information. Therefore, a hand-crafted summation strategy may limit the model’s ability to adaptively emphasize the most informative directional responses. To address these limitations, we propose Patch-MoE Mamba , a patch-ordered mixture-of-experts (MoE) state space architecture for medical image segmentation. Instead of scanning individual pixels in a raster-like order, we partition feature maps into local patches and perform directional scanning in a patch-ordered manner. This strategy keeps pixels within each patch consecutive in the sequence, thereby better preserving local spatial neighborhoods while still enabling long-range dependency modeling across patches. We further employ hierarchical patch sizes at different stages to capture both fine-grained boundary details and coarse semantic structures. In addition, we introduce an MoE-based directional fusion module to replace fixed summation. Specifically, the outputs of four directional Mamba scanners and their learnable concatenation are treated as five experts. A spatial-aware gating network computes location-dependent expert weights to adaptively fuse these directional responses. We also add a residual summation of the raw directional outputs to stabilize training and preserve strong directional signals. Experiments on five public polyp segmentation datasets and two skin lesion segmentation datasets show consistent improvements over CNN-, Transformer-, and Mamba-based baselines, demonstrating the effectiveness and generality of Patch-MoE Mamba. Fig. 1: (a) Overview of the proposed Patch-MoE Mamba architecture. (b) Structure of the Patch-MoE Visual State Space (VSS) block. (c) Structure of the Semantics and Detail Infusion (SDI) module. For simplicity, only the refinement of features at the third scale ( l=3 ) is shown. II Methods II-A Patch-MoE Mamba Architecture Fig. 1 (a) shows the overall architecture of Patch-MoE Mamba. It follows a U-Net-style design, consisting of a Mamba-based encoder, a semantics and detail infusion (SDI) module, and a decoder. Specifically, we replace the visual state space (VSS) block in VM-UNetV2 [ 12 ] with a new Patch-MoE VSS block that integrates a patch-ordered scanning mechanism (Section II-B ) and an adaptive MoE fusion module (Section II-C ) to form the encoder (Fig. 1 (b)). We then adopt the SDI module from U-Net v2 [ 15 ] (Fig. 1 (c)) to enhance the feature maps at each level by infusing semantic information from higher-level features and integrating finer details from lower-level features via the Hadamard product. Finally, we retain the decoder design of VM-UNetV2 [ 12 ] . II-B Patch-Ordered Scanning Existing Mamba-based models [ 10 , 11 , 12 ] typically treat a 2D feature map as a long 1D token sequence and apply pixel-level raster scanning. As illustrated in Fig. 2 (a), standard raster scanning traverses the grid row by row. While this formulation leverages the sequential modeling strength of state space models, it implicitly destroys local 2D structure: pixels that are adjacent in the scan order may be far apart spatially, and spatially neighboring pixels can be separated by a large sequence distance. For example, vertically adjacent pixels (positions 1 and 17 in the 16\times 16 grid) are separated by 16 steps in the 1D sequence. As a result, important local patterns such as lesion boundaries, fine surface structures, and local context around small or low-contrast objects are diluted or distorted in the sequence representation. This pixel-wise scanning thus tends to lose spatial coherence, which is undesirable for dense prediction tasks (e.g., image segmentation). To preserve spatial coherence during sequence construction, we introduce a patch-ordered scanning strategy in place of the raster-based flattening used by standard Vision Mamba modules. Given a feature map X_{l}\in\mathbb{R}^{C_{l}\times H_{l}\times W_{l}} and a patch size p , we partition the spatial grid into \lceil H_{l}/p\rceil\times\lceil W_{l}/p\rceil non-overlapping patches of at most p\times p pixels (Fig. 2 (b)). Within each patch, we enumerate all pixel locations (r,c) in row-major order and append their indices to a global index list before proceeding to the next patch. This process yields a permutation vector of length H_{l}\times W_{l} that reorders spatial positions but retains every pixel. In other words, the full-resolution feature grid is preserved and no pooling or token reduction occurs; only the visiting order changes. Compared to raster scanning, patch-ordered scanning ensures that all pixels inside a patch are mapped to consecutive positions in the sequence, thereby improving local spatial coherence while still allowing the state space model to capture long-range dependencies across patches. To increase spatial locality within contiguous segments of the sequence and to encode multi-scale context, we develop a hierarchical

Read the complete text