Hyper-Connections for Adaptive Multi-Modal MRI Brain Tumor Segmentation
Lokendra Kumar, Shubham Aggarwal · 2026
Read Hyper-Connections for Adaptive Multi-Modal MRI Brain Tumor Segmentation by Lokendra Kumar, Shubham Aggarwal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19844.txt.
Introduction
Kumar and Aggarwal ask whether brain tumor models use the right MRI scans for each finding. Radiologists lean on contrast scans for enhancing tumor and FLAIR for swelling. Most networks fuse all scans in a fixed way, whatever the patient or region. The authors build a probing toolkit that ablates each scan by region. It also tracks gate paths inside the network. They find that adaptive hyper-connections push models toward the clinically apt scans. This pattern recurs across three backbones, and accuracy also improves.
Catalog record
| CREATOR | Lokendra Kumar, Shubham Aggarwal |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44430 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.19844
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Hyper-Connections for Adaptive Multi-Modal MRI Brain Tumor Segmentation Lokendra Kumar, Shubham Aggarwal Abstract We present the first study of Hyper-Connections (HC) for volumetric multi-modal brain tumor segmentation, integrating them as a drop-in replacement for fixed residual connections across five architectures: nnU-Net, SwinUNETR, VT-UNet, U-Net, and U-Netpp. Dynamic HC consistently improves all 3D models on the BraTS 2021 dataset, yielding up to + 1.03 % +1.03\% mean Dice gain with negligible parameter overhead. Gains are most pronounced in the Enhancing Tumor sub-region, reflecting improved fine-grained boundary delineation. Modality ablation further reveals that HC-equipped models develop sharper sensitivity toward clinically dominant sequences, specifically T1ce for Tumor Core and Enhancing Tumor, and FLAIR for Whole Tumor, a behavior absent in fixed-connection baselines and consistent across all architectures. In 2D settings, improvements are smaller and configuration-sensitive, suggesting that volumetric spatial context amplifies the benefit of adaptive aggregation. These results establish HC as a simple, efficient, and broadly applicable mechanism for multi-modal feature fusion in medical image segmentation. 1 Introduction Brain tumors are among the most life-threatening neurological disorders, requiring precise diagnosis and treatment planning to improve patient survival and quality of life. Magnetic Resonance Imaging (MRI) is widely used for brain tumor assessment due to its ability to capture detailed soft tissue structures. However, manual delineation of tumor regions is time-consuming, subjective, and prone to inter-observer variability. Automated segmentation models have therefore become essential in modern clinical workflows, enabling accurate identification of tumor sub-regions such as Enhancing Tumor (ET), Tumor Core (TC), and Whole Tumor (WT). Reliable segmentation directly impacts downstream tasks including surgical planning, radiotherapy targeting, disease progression monitoring, and treatment response evaluation, making it a critical component in medical image analysis. The BraTS benchmark 11 ; 2 has standardized multi-modal brain tumor segmentation using four MRI sequences T1, T1ce, T2, and FLAIR, which provide complementary information about tumor structure and pathology. Over the years, a wide range of architectures have been developed for this task. For volumetric (3D) segmentation, models such as nnU-Net 8 , SwinUNETR 3 , and VT-UNet 13 have demonstrated strong performance by capturing spatial context and global dependencies. For slice-wise (2D) segmentation, architectures including U-Net 14 and U-Netpp 17 remain widely used due to their efficiency and effectiveness. Despite these differences, most approaches rely on fixed residual and skip connections for feature aggregation, which are inherently input-independent and limit the model’s ability to adapt to varying modality importance and tumor characteristics. In this work, we explore Hyper-Connections (HC) 18 , a recently proposed generalization of residual connections that enables dynamic, input-dependent feature aggregation. While HC has shown promising results in domains such as large language models and image generation and classification, it has not been investigated for medical image segmentation, particularly in volumetric and multi-modal settings. We extend HC to both 3D and 2D segmentation architectures and systematically evaluate its integration across nnU-Net, SwinUNETR, VT-UNet, U-Net, and U-Netpp. Our work demonstrates the effectiveness of HC in improving segmentation performance and provides insights into modality importance, showing how adaptive connections enable better utilization of multi-modal MRI information. 2 Related Work Brain tumor segmentation has been extensively studied through the BraTS challenge 11 ; 2 , with U-Net 14 establishing the encoder-decoder paradigm with skip connections as a dominant approach. nnU-Net 8 further advanced this direction by automating preprocessing, architecture configuration, and training strategies, and remains a strong convolutional baseline. Transformer-based approaches have introduced improved global context modeling, including UNETR 4 , which employs a transformer encoder with a convolutional decoder, SwinUNETR 3 , which leverages hierarchical shifted-window attention 9 , and VT-UNet 13 , which incorporates volumetric self-attention with 3D positional encoding. Despite these advances, all such architectures rely on fixed and non-adaptive connection mechanisms. Adaptive connection and attention mechanisms have been explored to improve feature representation and training dynamics. Residual connections 5 enable stable optimization of deep networks through identity mappings, while Squeeze-and-Excitation networks 7 introduce channel-wise adaptive reweighting. Attention gates 12 enhance skip connections by focusing on relevant spatial regions. Other approaches such as ReZero 1 and ResiDual 15 modify residual pathways to improve convergence and stability. However, these methods typically target specific connection types or architectural components. In contrast, the Hyper-Connection framework 18 provides a unified formulation that simultaneously adapts both depth-wise and width-wise connections, representing a strict generalization of prior approaches. To the best of our knowledge, this work presents the first evaluation of hyper-connections in volumetric medical image segmentation. Multi-modal and multi-scale feature fusion is central to MRI-based segmentation. U-Netpp 17 introduced densely nested skip connections to bridge the semantic gap between encoder and decoder representations, while attention-based methods 12 learn to emphasize spatially relevant features. Prior work has also explored modality-specific weighting strategies for BraTS segmentation 16 . Hyper-connections generalize these ideas by enabling adaptive aggregation across all connections in the network, conditioned on global input statistics at each layer, thereby providing a unified and flexible mechanism for multi-modal feature integration. 3 Background and Motivation 3.1 From Language Models to Volumetric Medical Imaging Hyper-Connections (HC) were originally proposed in the context of large language model pre-training, where the primary challenge is to maintain stable gradient flow across very deep transformer architectures operating on sequential token embeddings 18 . In that setting, HC improves optimization by learning adaptive depth-wise aggregation, thereby avoiding limitations associated with fixed residual formulations such as Pre-Norm and Post-Norm variants. In contrast, volumetric medical image segmentation presents a fundamentally different problem. Instead of discrete tokens, the input consists of continuous-valued three-dimensional voxel grids, and the output requires dense, voxel-level predictions with high spatial precision. Furthermore, MRI-based segmentation is inherently multi-modal, with sequences such as T1, T1ce, T2, and FLAIR providing complementary but unevenly informative signals. The task is further complicated by the nested anatomical structure of tumor sub-regions and the reliance on encoder-decoder architectures with skip connections that preserve spatial correspondence. These differences raise an important question: whether the benefits of HC extend beyond sequence modeling to structured, multi-modal, and spatially grounded tasks such as medical image segmentation. 3.2 Limitations of Fixed Connections in Multi-Modal MRI Segmentation Most existing architectures rely on fixed residual connections of the form x_{l+1}=x_{l}+F(x_{l}), (1) where the contribution of the residual branch is constant and independent of the input 5 . Similarly, skip connections in encoder-decoder architectures combine features using fixed rules such as concatenation or summation. While effective in general settings, such fixed aggregation introduces several limitations in multi-modal MRI segmentation. First, these connections are modality-blind, meaning they cannot dynamically adjust the contribution of different MRI sequences, even though modalities such as FLAIR and T1ce capture distinct biological characteristics. Second, they impose depth-invariant weighting, treating shallow and deep features equally across all spatial regions, despite the fact that different tumor sub-regions require different levels of semantic and textural information. Third, they do not account for the hierarchical structure of tumor regions, where Enhancing Tumor is contained within Tumor Core, which is further contained within Whole Tumor. Finally, fixed connections are inherently patient-invariant, applying the same aggregation strategy to all inputs, even though tumor appearance varies significantly across patients. These limitations motivate the need for adaptive connection mechanisms that can adjust feature aggregation based on input characteristics. 3.3 Adaptive Multi-Modal Feature Aggregation in HC-Powered Architectures