RTFDNet: Fusion-Decoupling for Robust RGB-T Segmentation

Kunyu Tan, Mingjian Liang † · 2026

Read RTFDNet: Fusion-Decoupling for Robust RGB-T Segmentation by Kunyu Tan, Mingjian Liang † (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-09149.txt.

Introduction

Kunyu Tan and Mingjian Liang present RTFDNet, a three-branch network for RGB-thermal segmentation that stays robust when sensor signals partly go missing. Older methods stress balance between the two modes yet degrade badly under such loss. Their design unifies fusion and decoupling in one encoder-decoder, with gated channel exchange and light spatial attention sharing cues. Two regularization terms keep unimodal paths strong without harming the fused stream, allowing efficient standalone inference.

Catalog record

CREATORKunyu Tan, Mingjian Liang †
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31972
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.09149

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RTFDNet: Fusion-Decoupling for Robust RGB-T Segmentation Kunyu Tan, Mingjian Liang † Abstract RGB-Thermal (RGB-T) semantic segmentation is essential for robotic systems operating in low-light or dark environments. However, traditional approaches often overemphasize modality balance, resulting in limited robustness and severe performance degradation when sensor signals are partially missing. Recent advances such as cross-modal knowledge distillation and modality-adaptive fine-tuning attempt to enhance cross-modal interaction, but they typically decouple modality fusion and modality adaptation, requiring multi-stage training with frozen models or teacher–student frameworks. We present RTFDNet, a three-branch encoder–decoder that unifies fusion and decoupling for robust RGB-T segmentation. Synergistic Feature Fusion (SFF) performs channel-wise gated exchange and lightweight spatial attention to inject complementary cues. Cross-Modal Decouple Regularization (CMDR) isolates modality-specific components from the fused representation and supervises unimodal decoders via stop-gradient targets. Region Decouple Regularization (RDR) enforces class-selective prediction consistency in confident regions while blocking gradients to the fusion branch. This feedback loop strengthens unimodal paths without degrading the fused stream, enabling efficient standalone inference at test time. Extensive experiments demonstrate the effectiveness of RTFDNet , showing consistent performance across varying modality conditions. Our implementation will be released to facilitate further research. Our source code are publicly available at https://github.com/curapima/RTFDNet . I Abstract II Introduction Multimodal perception has become instrumental in Embodied-AI [ 15 , 35 , 28 ] , advanced robotics [ 23 , 22 , 26 ] and autonomous vehicles [ 6 , 32 , 37 ] . Particularly, the fusion of RGB and Thermal (RGB-T) imagery leverages the complementary characteristics of these sensors, combining the rich texture from RGB cameras with the robustness of Thermal imaging in poor illumination or adverse weather, enabling reliable robotic operations in darkness [ 9 ] , enhancing autonomous driving under inclement conditions [ 19 ] , and facilitating specialized tasks such as cave exploration [ 26 ] . The success of these applications hinges on sophisticated fusion strategies that integrate information from both modalities to achieve performance exceeding that of any single sensor [ 36 , 2 , 16 , 25 , 39 ] . Fig. 1: Architecture comparison of training paradigms for RGB–Thermal segmentation under missing modalities. (A) Two-stage knowledge distillation trains a multimodal teacher then distills to per-case students. (B) Modal-adaptation fine-tuning freezes the base and updates lightweight adapters to accommodate a dropped modality. (C) Our unified robustness training jointly optimizes RGB/Thermal encoders, a fusion branch, and multi-branch decoders with bidirectional consistency. At inference, we need to load only the corresponding encoder and decoder parameters. However, a fundamental assumption underpins the vast majority of state-of-the-art models, namely that all sensor inputs are consistently available and perfectly aligned. This assumption is fragile in real-world deployments, where sensors are prone to failure from hardware malfunctions, signal corruption, or environmental interference. In such cases, how does performance degrade? As shown in Table I , we observe a disquieting trend that when one modality is lost, the performance of leading RGB-T segmentation models [ 30 , 37 ] drops sharply, often falling below that of a system trained on a single modality from the start. This reveals that current multimodal systems lack the robustness required for dependable real-world operation in the face of sensor failures. Traditional approaches [ 10 , 5 ] distill the multimodal knowledge of an RGB-T teacher into a single-modal student (RGB or Thermal). This paradigm hinders efficient training because it typically requires training separate students for each missing-modality scenario. To improve the efficiency of teacher-student pipelines, more recent approaches aim to build robustness directly into the model or its training process. These methods largely fall into two categories. The first is post hoc, parameter-efficient adaptation [ 24 ] as shown in Fig. 1 (A). In this paradigm, a large pre-trained multimodal model is kept frozen, and lightweight adapters are trained specifically for each missing-modality scenario. The second is to integrate robustness into the training objective itself [ 25 ] as illustrated in Fig. 1 (B), where a single model is trained from the start to be resilient to missing inputs (e.g., via self-distillation with complementary masking). However, despite these advances, significant limitations remain. (i) Insufficient feature fusion attention. Adapter-based robustness freezes the multimodal backbone and trains small, scenario-specific adapters. Because the fusion path gets little to no supervision, adapters hinge on a fixed fused representation, so unimodal branches gain little and remain weak when that representation is suboptimal. (ii) Insufficient decoupling. Single modality inputs still traverse the fused path, preventing standalone RGB only or T only inference needed for sensor dropouts, safety, and efficient fallback. TABLE I: Comparison on RGB-T segmentation with modality drops, (including signal loss, image corruption, and signal instability). Methods RGB-T RGB drop Thermal drop \text{mIoU}\,\uparrow \text{mIoU}\,\uparrow Diff \downarrow \text{mIoU}\,\uparrow Diff \downarrow RTFNet(ResNet152) 53.2 37.3 -15.9 24.57 -28.63 EAEFNet(ResNet152) 58.95 35.23 -23.72 41.72 -17.23 CMXNet(segformer-b4) 59.77 53.55 -6.22 35.46 -24.31 StitchFusion(segformer-b2) 58.04 48.78 -9.26 41.42 16.62 CRM(swin-T) 59.1 50.98 -8.12 50.22 -8.88 Ours(segformer-b2) 58.97 55.12 -3.85 53.2 -5.77 To tackle these limitations, in this paper we propose RTFDNet, which combines the F usion and D ecouple strategy in a three-branch architecture, as illustrated in Fig. 1 (C). Our goal is a representation-level reversible pipeline: forward complementary fusion strengthens the fused stream, while reverse decoupling preserves modality-recoverable components to guide each unimodal branch for standalone fallback. The key design includes three modules. Synergistic Feature Fusion (SFF) enriches each single-modality branch with complementary cues through selective cross-modal interaction, and Cross-modal Decouple Regularization (CMDR) explicitly separates modality-specific features from shared ones. Region Decouple Regularization (RDR) ensures that all branches produce semantically consistent outputs. These components create a unified feedback loop that jointly optimizes the RGB and Thermal encoders, the fusion branch, and their respective decoders. Specifically, at the inference stage, one simply loads the encoder and decoder corresponding to the available modality. This design not only preserves the high accuracy of the full-modality system but also markedly improves its robustness with missing or corrupted inputs. Our contributions are threefold: • A novel fusion and decoupling method that combines RGB and Thermal information and preserves modality-recoverable components in the fused representation for reverse guidance . • An efficient and inference-capable, parameter-separable, three-branch encoder-decoder network assembled with the proposed feature fusion and decoupling strategy. • State-of-the-art performance on three RGB-T semantic segmentation benchmark datasets. The remainder of this paper is organized as follows. Section II reviews related work. Section III details the proposed framework and fusion strategy. Section IV presents both quantitative and qualitative evaluations. Section V concludes the study. Fig. 2: Diagram of encoder–decoder model where SFF executes modality fusion, and CMDR/RDR decouple the fused features to regularize and guide each single-modality branch. (The robotic platform is a conceptual illustration.) III related work III-A Feature Fusion in RGB-T Semantic Segmentation Leveraging Thermal images to complement RGB images has shown great improvements in semantic segmentation accuracy. Previous modality fusion methods such as [ 6 , 30 , 31 ] integrated through direct feature addition, verifying the effectiveness of the feature fusion strategy. Recent work focuses on enhancing finer-grained feature fusion to facilitate more comprehensive cross-modal interaction. Methods such as [ 38 , 16 , 13 , 42 , 3 , 41 ] promote interplay between modalities through more elaborate, dedicated fusion modules. Other

Read the complete text