Report: Weather-Aware Object Detection Transformer for Domain Adaptation
Soheil Gharatappeh, Salimeh Yasaei Sekeh, Vikas Dhiman · 2025
Read Report: Weather-Aware Object Detection Transformer for Domain Adaptation by Soheil Gharatappeh, Salimeh Yasaei Sekeh, Vikas Dhiman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10877.txt.
Introduction
Soheil Gharatappeh, Salimeh Sekeh, and Vikas Dhiman try three ways to make a real-time detection transformer hold up in fog. Their 2025 report starts from a known weakness: the RT-DETR family detects objects well in clear weather but degrades when fog rolls in. Each of their designs attacks the domain gap between clear and foggy images differently.
One approach distills fog-invariant features from a teacher network. A second adds fog-sensitive scaling inside the attention mechanism, fed by a parallel foggy stream. A third fuses clear and foggy features in a dual-stream encoder. None of the three consistently beat the plain baseline. The authors analyze why, offering guidance for later weather-aware detection work.
Catalog record
| CREATOR | Soheil Gharatappeh, Salimeh Yasaei Sekeh, Vikas Dhiman |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10877
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Report: Weather-Aware Object Detection Transformer for Domain Adaptation Soheil Gharatappeh, Salimeh Yasaei Sekeh, Vikas Dhiman Abstract RT-DETRs have shown strong performance across various computer vision tasks but are known to degrade under challenging weather conditions such as fog. In this work, we investigate three novel approaches to enhance RT-DETR robustness in foggy environments: (1) Domain Adaptation via Perceptual Loss, which distills domain-invariant features from a teacher network to a student using perceptual supervision; (2) Weather Adaptive Attention, which augments the attention mechanism with fog-sensitive scaling by introducing an auxiliary foggy image stream; and (3) Weather Fusion Encoder, which integrates a dual-stream encoder architecture that fuses clear and foggy image features via multi-head self and cross-attention. Despite the architectural innovations, none of the proposed methods consistently outperform the baseline RT-DETR. We analyze the limitations and potential causes, offering insights for future research in weather-aware object detection. 1 Introduction Object detection algorithms have witnessed remarkable advancements in recent years, evolving from computationally expensive two-stage approaches [ 20 , 8 ] to highly efficient single-stage architectures [ 19 , 18 ] . Early methods such as Faster R-CNN [ 20 ] demonstrated strong performance but suffered from high latency due to their region proposal mechanisms. The advent of single-stage detectors like YOLO and SSD revolutionized real-time object detection by eliminating the need for proposal generation, relying instead on direct regression and classification. However, these architectures heavily incorporate hand-crafted components such as anchor boxes and non-maximum suppression, which impose constraints on their adaptability to diverse environments. More recently, transformer-based architectures have emerged as a powerful alternative to CNN-based networks [ 7 ] , demonstrating superior performance on various computer vision benchmarks. Models such as DETR [ 3 ] and its variants [ 29 , 17 ] leverage the self-attention mechanism to model long-range dependencies and learn object representations with greater flexibility. Unlike CNNs, transformers do not rely on hand-crafted design choices, making them more adaptable to different visual contexts [ 7 ] . However, despite these advantages, transformer-based detectors remain highly sensitive to image degradation caused by adverse weather conditions, such as fog [ 28 ] . In real-world scenarios, fog significantly decreases image quality by reducing contrast, obscuring fine details, and introducing scattering effects, which severely impact the feature extraction capability of these models. Unsupervised Domain Adaptation (UDA) has been extensively studied to address domain shifts arising from variations between a labeled source domain and an unlabeled target domain [ 5 , 16 , 30 ] . UDA techniques typically focus on aligning feature distributions between domains to facilitate knowledge transfer. This is often achieved through adversarial learning [ 10 ] , self-supervised objectives [ 34 ] , or statistical feature alignment [ 32 ] . However, these methods primarily emphasize the alignment of cross-domain features, which can inadvertently lead to the loss of discriminatory information. Specifically, in adverse weather conditions, UDA approaches may struggle to disentangle domain-specific artifacts (e.g., fog-induced distortions) from essential object features, leading to suboptimal detection performance. To overcome these limitations, we explore alternative strategies that emphasize feature robustness without relying solely on domain alignment. Specifically, we propose integrating perceptual loss into transformer-based detectors, specifically RT-DETRs, to enhance robustness. Originally introduced for tasks like image synthesis and style transfer [ 15 ] , perceptual loss preserves high-level semantic features by minimizing discrepancies in deep feature space rather than at the pixel level. By integrating this loss into a knowledge distillation based pipeline [ 14 ] , we aim to encourage the model to learn domain-invariant yet discriminative representations, helping to retain critical object features despite fog-induced degradation. In addition to feature-level alignment, we explore a weather-adaptive attention mechanism that enhances the model’s ability to modulate attention based on the context of the fog. This approach introduces an auxiliary stream that processes a foggy version of the input image, producing scalar weights that adjust the attention computation in the transformer encoder. This dynamic modulation allows the model to better emphasize relevant features even when visibility is reduced. Finally, we propose a weather fusion encoder that operates on both clear and foggy versions of the input image in parallel. Each stream undergoes self-attention refinement, and their features are subsequently fused through a cross-attention module. This design enables the model to integrate complementary cues from both domains, promoting more robust and adaptive object detection in visually challenging foggy scenes. The main contributions of this paper are: • We proposed domain adaptation through perceptual loss and knowledge transfer through a teacher network. • We proposed FogAwareAttention, an attention mechanism that adapts RT-DETR encoder values to foggy weather by incorporating the foggy image pipeline. • We proposed a Weather Fusion Encoder where a cross-attention mechanism is used to fuse the clear image and foggy image embeddings. The remainder of this paper is structured as follows. Section 2 reviews related work on object detection, domain adaptation, perceptual loss, and RT-DETRs. Section 3 introduces our proposed method and details the integration of three proposed methods. Section 4 presents experimental results, including comparisons with existing methods under foggy conditions. Finally, Section 5 concludes the paper and discusses potential future research directions. 2 Related Work 2.1 Object Detection Using Transformers Deep convolutional neural networks (CNNs) have become the standard approach for object detection, with two-stage methods such as Faster R-CNN [ 20 ] and one-stage approaches like YOLO and SSD [ 19 , 18 ] demonstrating strong performance on large-scale benchmarks. Despite their success, these methods rely heavily on handcrafted components such as non-maximum suppression (NMS), Intersection over Union (IoU) thresholds, and anchor box design. As a result, they often require extensive fine-tuning and cannot be trained in an end-to-end fashion. Transformers, originally designed for sequence modeling in natural language processing, have recently been adapted for vision tasks. Vision Transformers (ViTs) showed promising results but were initially limited by their high data requirements, often underperforming CNNs when trained on smaller datasets. DEtection TRansformers (DETRs) addressed this by combining a CNN backbone with a transformer-based encoder-decoder structure, leveraging multi-head self-attention to model global context effectively [ 3 ] . This made DETRs the first end-to-end object detection framework based on transformers. Real-Time DETRs (RT-DETRs) further improved efficiency and accuracy by introducing a multi-scale feature pyramid, an Efficient Hybrid Encoder, and a Cross-Scale Feature Fusion (CSFF) block [ 33 ] . These components enhanced feature representation at multiple resolutions and enabled the detection of small objects—an area where traditional transformers often struggled. RT-DETR’s streamlined design allows for real-time inference while preserving the benefits of transformer-based modeling. 2.2 Domain Adaptation for Object Detection Unsupervised Domain Adaptation (UDA) has been extensively studied to address performance degradation caused by distribution shifts between a labeled source domain and an unlabeled target domain. Existing UDA methods for object detection are primarily categorized into two levels: domain-level alignment and category-level alignment. Domain-level UDA focuses on reducing the overall distribution discrepancy between the source and target domains. This is typically achieved by aligning global feature distributions across domains using statistical measures such as Maximum Mean Discrepancy [ 31 , 26 ] ] and Correlation Alignment [ 23 ] . These methods attempt to map source and target features into a shared feature space by minimizing distributional divergence,