Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities

Peibo Song, Xiaotian Xue 2 {2}, Jinshuo Zhang, Zihao Wang, Jinhua Liu, Shujun Fu 1 {1}, Fangxun Bao, Si Yong Yeo · 2026

Read Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities by Peibo Song, Xiaotian Xue 2 {2}, Jinshuo Zhang, Zihao Wang, Jinhua Liu, Shujun Fu 1 {1}, Fangxun Bao, Si Yong Yeo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-22177.txt.

Introduction

Song and seven colleagues propose UniME, a two-stage method for brain tumor segmentation with missing MRI modalities. Stage 1 pretrains a single ViT Uni-Encoder with masked image modeling, building a unified representation robust to missing modalities. Stage 2 adds special CNN Multi-Encoders to extract high-resolution, multi-scale, fine-grained features. Fusing these features with the global picture yields precise segmentations. On BraTS 2023 and BraTS 2024, UniME outperforms previous methods under incomplete multi-modal scenarios.

Catalog record

CREATORPeibo Song, Xiaotian Xue 2 {2}, Jinshuo Zhang, Zihao Wang, Jinhua Liu, Shujun Fu 1 {1}, Fangxun Bao, Si Yong Yeo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39659
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.22177

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities Peibo Song, Xiaotian Xue 2 {2}, Jinshuo Zhang, Zihao Wang, Jinhua Liu, Shujun Fu 1 {1}, Fangxun Bao, Si Yong Yeo Abstract Multimodal MRI offers complementary information for brain tumor segmentation, but clinical scans often lack one or more modalities, which degrades segmentation performance. In this paper, we propose UniME ( U ni-Encoder M eets Multi- E ncoders), a two-stage heterogeneous method for brain tumor segmentation with missing modalities that reconciles the trade-offs among fine-grained structure capture, cross-modal complementarity modeling, and exploitation of available modalities. The idea is to decouple representation learning from segmentation via a two-stage heterogeneous architecture. Stage 1 pretrains a single ViT Uni-Encoder with masked image modeling to establish a unified representation robust to missing modalities. Stage 2 adds modality-specific CNN Multi-Encoders to extract high-resolution, multi-scale, fine-grained features. We fuse these features with the global representation to produce precise segmentations. Experiments on BraTS 2023 and BraTS 2024 show that UniME outperforms previous methods under incomplete multi-modal scenarios. The code is available at https://github.com/Hooorace-S/UniME . 1 1 footnotetext: Corresponding authors. 1 Introduction Accurate brain tumor segmentation from multi-modal MRI is clinically important 4 . Since different modalities provide complementary information, high-quality segmentation requires capturing fine anatomical details and effectively leveraging cross-modal complementarity 43 ; 59 . However, in clinical practice, one or more MRI modalities are often missing 28 . For example, T2-weighted MRI is usually unavailable due to its susceptibility to artifacts 21 . As illustrated in Fig. 1 (a), such incomplete modality settings degrade segmentation performance 9 , raising two challenges: (i) how to model cross-modal complementarity to mitigate the impact of missing modalities, and (ii) how to fully exploit the information present within the available modalities. Figure 1 : (a) Segmentation under missing MRI modalities. , , and indicate the necrotic tissue, enhancing tumor, and edema regions, respectively. “Missing” denotes unavailable modality. (b) Trade-offs among three design goals. Existing methods typically trade off among fine-grained structure capture, cross-modal complementarity modeling, and effective exploitation of available modalities. UniME reconciles these trade-offs by introducing a two-stage heterogeneous design, which decouples representation learning from segmentation. Recent methods mainly fall into two families. The first aims to maximize cross-modal complementarity, via modality synthesis 29 ; 65 ; 58 ; 55 ; 26 or knowledge distillation 49 ; 3 ; 34 ; 51 . However, synthesis usually fails to recover fine details, and distillation pipelines are often complex and difficult to scale. The second family redesigns architectures to better use the available modalities 23 ; 15 ; 62 ; 39 ; 46 ; 63 ; 33 , but the limited receptive fields and high computational cost hinder the capture of global, cross-modal semantics. As shown in Fig. 1 (b), an ideal method should capture fine-grained structures, model cross-modal complementarity, and effectively exploit the available modalities, but achieving these goals simultaneously remains challenging. The rise of self-supervised representation learning offers a new perspective 7 ; 24 ; 53 ; 50 ; 19 . Large-scale self-supervised Vision Transformer (ViT) models have been shown to effectively capture global semantics and model cross-modal relationships 32 ; 2 ; 27 ; 35 ; 19 ; 25 ; 40 . This suggests a new route to capture global semantics and model cross-modal complementarity. However, in medical image segmentation, a pure ViT network without convolutional inductive biases often fails to meet the pixel-level precision required to capture fine anatomical structures 45 . Moreover, relying solely on a unified encoder is insufficient to fully exploit modality-specific information, limiting segmentation accuracy under incomplete modality scenarios. To this end, we propose UniME , a two-stage heterogeneous segmentation method that explicitly decouples representation learning from segmentation, effectively modeling cross-modal complementarity through masked self-supervised learning, and fully utilizing available modalities while capturing fine-grained structures via a heterogeneous architecture design. In Stage 1 , we pretrain a single ViT Uni-Encoder with masked self-supervision that combines modality-level and patch-level masking to learn a unified representation robust to missing modalities. To improve semantic generalization under limited data, the Uni-Encoder incorporates spatial priors through effective positional embedding strategies. In Stage 2 , we add parallel modality-specific CNN Multi-Encoders to extract high-resolution, multi-scale, fine-grained features, which enrich Uni-Encoder features with multi-scale detail. To ensure smooth fine-tuning, we apply layer-wise learning rate decay (LLRD), which helps preserve the pretrained semantic knowledge captured by the Uni-Encoder. In summary, our main contributions are: • We introduce UniME , to our knowledge, the first two-stage heterogeneous method for brain tumor segmentation with missing modalities, effectively reconciling the trade-offs among fine-grained structure capture, cross-modal complementarity modeling, and effective exploitation of available modalities. • To model cross-modal complementarity, UniME pretrains a Uni-Encoder with masked self-supervision to obtain unified representations robust to absent modalities. • To exploit available modalities and capture fine-grained details, UniME employs a heterogeneous architecture that introduces Multi-Encoders to enrich the Uni-Encoder with multi-scale, fine-grained features. • Experiments on BraTS 2023 and BraTS 2024 validate UniME’s effectiveness, particularly in scenarios with missing modalities. 2 Related Work Incomplete Multimodal Tumor Segmentation. Incomplete data are common in practice 64 ; 48 ; 42 . In this work, we study brain tumor segmentation with missing modalities. Compared with standard brain tumor segmentation 57 ; 36 ; 56 ; 6 ; 8 ; 22 ; 45 , the missing-modality setting is both more practical and more challenging than the standard setting. Limitations in modality synthesis 29 ; 65 ; 58 ; 55 ; 26 and in knowledge distillation 49 ; 3 ; 34 ; 51 have shifted attention to architectural designs that better use the available modalities. Havaei et al. 23 introduced HeMIS, which uses four parallel encoders and a shared decoder for multimodal fusion. This established a baseline architecture adopted by much subsequent work. Later studies largely retained the HeMIS framework and augmented it with stronger modules or fusion strategies 15 ; 62 ; 39 ; 46 ; 63 ; 33 . Despite these advances, most methods follow the HeMIS paradigm: extract features with multiple independent CNN encoders and fuse them downstream. Even with Transformer-based or other global context fusion 62 ; 39 ; 15 ; 63 ; 33 , the upstream CNN encoders impose local receptive field limits. This dependency forms a performance ceiling for cross-modal feature fusion and is a key bottleneck to further progress. Representation Learning. In recent years, self-supervised representation learning has reshaped computer vision 7 ; 24 ; 53 ; 50 ; 19 . Masked pretraining methods show that a single ViT encoder can learn transferable representations from unlabeled data 24 ; 53 ; 50 ; 19 . Moreover, recent studies show that large-scale, extensively pretrained ViTs can both capture high-level semantics and model cross-modal relationships 32 ; 2 ; 27 ; 35 ; 19 ; 25 ; 40 . For brain tumor segmentation with missing modalities, several methods also adopt two-stage masked modeling. Liu et al. 31 proposed M 3 AE, which uses a symmetric U-Net for both pretraining and fine-tuning. However, its full CNN decoder and skip connections diminish the encoder’s effectiveness in representation learning 53 ; 60 . Similarly, Zhang et al. 61 introduced M 2 SegMamba, which applies semantic-level masking while retaining

Read the complete text