BTI-Net: Bidirectional Decoder-Level Task Interaction via Uncertainty-Aware Gating for Multi-Task Medical Image Analysis

Abdullah Al Shafi, Md Kawsar Mahmud Khan Zunayed, Safin Ahmmed, Sk Imran Hossain, Engelbert Mephu Nguifo, [4pt] Khulna University of Engineering & Technology, Bangladesh University Clermont Auvergne, France · 2026

Read BTI-Net: Bidirectional Decoder-Level Task Interaction via Uncertainty-Aware Gating for Multi-Task Medical Image Analysis by Abdullah Al Shafi, Md Kawsar Mahmud Khan Zunayed, Safin Ahmmed, Sk Imran Hossain, Engelbert Mephu Nguifo, [4pt] Khulna University of Engineering & Technology, Bangladesh University Clermont Auvergne, France (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29102.txt.

Introduction

Al Shafi, Khan Zunayed, Ahmmed, Hossain, Mephu Nguifo and colleagues build BTI-Net, a network that learns medical image segmentation and classification together, letting the two tasks talk at every stage. Most multi-task designs share only the encoder, so each decoder rebuilds its output from private features. Boundary cues useful for classification and semantic hints useful for segmentation are then permanently lost to the other branch.

BTI-Net adds two-way communication at every decoder level through task interaction modules. Boundary context enters the classification branch through a gate, while global semantic priors scale the segmentation decoder, moving from coarse meaning to fine boundaries across four resolutions. An uncertainty proxy module gates each interaction per image and level, using alignment, complexity and confidence signals, with no extra annotations. On ultrasound, skin and brain MRI benchmarks, the net beat rival baselines on segmentation overlap and classification accuracy.

Catalog record

CREATORAbdullah Al Shafi, Md Kawsar Mahmud Khan Zunayed, Safin Ahmmed, Sk Imran Hossain, Engelbert Mephu Nguifo, [4pt] Khulna University of Engineering & Technology, Bangladesh University Clermont Auvergne, France
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29102

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

BTI-Net: Bidirectional Decoder-Level Task Interaction via Uncertainty-Aware Gating for Multi-Task Medical Image Analysis Abdullah Al Shafi, Md Kawsar Mahmud Khan Zunayed, Safin Ahmmed, Sk Imran Hossain, Engelbert Mephu Nguifo, [4pt] Khulna University of Engineering & Technology, Bangladesh University Clermont Auvergne, France Abstract Jointly learning to segment and classify medical images demands cross-task synergy, yet encoder-sharing architectures limit decoder reconstruction to task-private representations, permanently discarding the boundary cues and semantic priors each branch could supply to the other. This work introduces BTI-Net, which establishes bidirectional communication at every decoder level through two parallel pathways via Task Interaction Modules (TIM). Spatial boundary context is gated into the classification branch, while global semantic priors multiplicatively modulate the decoder, with refined features propagating progressively from coarse semantics to fine boundary detail across all four decoder resolutions. Since cross-task interaction is not equally reliable for every input, Uncertainty Proxy Attention (UPA) gates each TIM output per instance and per level using three signals that capture cross-task alignment, scene complexity, and prediction confidence, without external annotations or additional inference passes. Experiments on three medical benchmarks spanning ultrasound, dermoscopy, and brain MRI demonstrate consistent improvements in segmentation IoU and classification accuracy over both encoder-sharing and decoder-interaction baselines. Ablation confirms adaptive gating contributes + + 2.36 IoU over fixed bidirectional interaction, and classification accuracy improves by up to + + 2.26 points over the strongest multi-task baseline. UPA’s uncertainty proxies serve as reliable single-pass task-failure signals without the overhead of stochastic sampling. Code: https://github.com/C-loud-Nine/BTI-Net_MTL † † footnotetext: † Equal contribution. 1 Introduction MTL enables a single model to address multiple related tasks jointly, with the dominant approach coupling a shared encoder to independent decoder heads. While efficient, this design restricts all inter-task communication to a coarse bottleneck, after which the decoders diverge independently. Segmentation recovers precise boundaries that could disambiguate classification; classification accumulates global semantic context that could resolve ambiguous spatial regions. Both signals are lost once the decoders diverge, a gap that is especially consequential in medical imaging where datasets are small, pathology is heterogeneous, and complementary task signals carry real diagnostic value. Recent decoder-interaction methods partially address this. MTI-Net [ 23 ] shows decoder-level interaction outperforms encoder-sharing, but propagates information in one direction with fixed per-sample blending. DenseMTL [ 15 ] adds bidirectionality but applies uniform blending regardless of scene complexity. In medical imaging, per-instance adaptability is not optional: the appropriate interaction strength varies fundamentally across pathology types, imaging conditions, and lesion morphology. Addressing both limitations, BTI-Net introduces Task Interaction Modules (TIM) for bidirectional decoder-level communication, paired with Uncertainty Proxy Attention (UPA) for per-instance adaptive gating. TIM establishes bidirectional communication at every decoder level, from coarsest semantics to finest boundary detail: spatially aggregated boundary cues are injected into the classification vector through a gated residual (Seg \rightarrow Clf), while global semantic priors are multiplicatively broadcast across the decoder map (Clf \rightarrow Seg), with refined features carried forward progressively across all four resolutions. UPA gates each TIM output per instance and per level using three signals that capture cross-task alignment, scene complexity, and prediction confidence. Independent sigmoid weights allow both tasks to be simultaneously enhanced when interaction is reliable; on easy samples the gate remains conservative, preserving the unperturbed representations. A difficulty-aware loss aligns the gate with per-sample task performance, requiring no external annotations and no additional inference passes. Evaluated on three publicly available benchmarks spanning ultrasound, dermoscopy, and brain MRI, BTI-Net advances both segmentation and classification consistently across modalities. Ablation confirms that adaptive gating contributes + 2.36 IoU over fixed bidirectional interaction, and the complete model improves classification accuracy by up to + 2.26 points over the strongest multi-task baseline. The main contributions are: 1. TIM: bidirectional decoder-level interaction between segmentation and classification via channel-wise gated operations, with refined spatial features propagated progressively across all four decoder resolutions. 2. UPA: a per-instance, per-level adaptive gate using three interpretable uncertainty signals, trained from per-sample difficulty scores derived automatically from task performance, without external annotations or Bayesian overhead. 3. Cross-modality validation on three diverse medical benchmarks, including, to the best of our knowledge, the first decoder-interaction MTL evaluation on BRISC. 2 Related Work 2.1 Multi-Task Learning Architectures MTL architectures divide broadly into encoder-sharing and decoder-interaction approaches. Encoder-sharing methods [ 4 , 14 , 8 ] couple a common feature extractor to task-specific heads that diverge during upsampling, leaving cross-task signals untapped after the shared bottleneck. Several works introduce cross-task connections during upsampling. PAD-Net [ 27 ] distills auxiliary task predictions into final tasks unidirectionally, establishing the pattern of using intermediate task outputs to guide final predictions. MTI-Net [ 23 ] models interactions at every backbone scale via distillation units, showing decoder-level interaction outperforms encoder-sharing; however, the flow is strictly unidirectional with identical blending applied to every sample. DenseMTL [ 15 ] adds bidirectional pairwise cross-task attention in decoder exchange blocks, yet applies a fixed blending weight regardless of scene complexity. Transformer-based methods such as InvPT [ 28 ] and TaskPrompter [ 29 ] model dependencies through global self-attention across task tokens; because they operate on joint token sets rather than explicit directed pathways, they do not provide the separate per-task gating that characterizes bidirectional decoder interaction. Three limitations persist across all prior decoder-interaction methods: attention costs that grow with spatial resolution make them poorly suited to small medical datasets; fixed blending ignores per-instance variability in cross-task reliability; and none has been evaluated on medical imaging. BTI-Net addresses all three: TIM’s cost scales with channel dimension rather than spatial resolution, UPA provides per-instance adaptive gating, and the method is validated across three medical modalities, as summarized in Table 1 . Table 1 : Positioning of BTI-Net against representative MTL architectures. Decoder : cross-task connections via explicit directed pathways; Bidir. : both tasks update each other at the same resolution; Per-inst. : interaction strength adapts per sample; Medical : validated on medical imaging. Method Decoder Bidir. Per-inst. Medical MTAN [ 14 ] ✗ ✗ ✗ ✓ PAD-Net [ 27 ] ✓ ✗ ✗ ✗ MTI-Net [ 23 ] ✓ ✗ ✗ ✗ MTANet [ 13 ] ✗ ✗ ✗ ✓ MTL-OCA [ 9 ] ✗ ✗ ✗ ✓ DenseMTL [ 15 ] ✓ ✓ ✗ ✗ InvPT [ 28 ] ✓ ✗ ✗ ✗ DynaShare [ 18 ] ✗ ✗ ✓ ✗ BTI-Net ✓ ✓ ✓ ✓ 2.2 Uncertainty in Multi-Task Learning Uncertainty modelling in MTL has focused primarily on task-level loss balancing. Kendall et al. [ 12 ] proposed weighting losses by homoscedastic task uncertainty, while GradNorm [ 6 ] and PCGrad [ 30 ] balance tasks by normalising or projecting conflicting gradients. MTANet [ 13 ] and MTL-OCA [ 9 ] extend these ideas with attention-based dynamic weighting. All these approaches operate at the task level—scaling each task’s loss contribution—and cannot modulate cross-task feature exchange strength for individual inputs. Monte Carlo dropout offers per-sample uncertainty estimates but requires multiple stochastic forward passes, prohibitive for dense prediction. The closest approach to instance-level feature adaptation is DynaShare [ 18 ] , which learns a per-input gating policy over encoder layers; however, it gates layer execution rather than cross-task feature exchange, requires a separate policy network, and has not been applied to dense medical analysis. UPA gates cross-task enhancement per instance

Read the complete text