DMAT: An End-to-End Framework for Joint Atmospheric Turbulence Mitigation and Object Detection

Paul Hill, Alin Achim, Dave Bull, Nantheera Anantrasirichai · 2025

Read DMAT: An End-to-End Framework for Joint Atmospheric Turbulence Mitigation and Object Detection by Paul Hill, Alin Achim, Dave Bull, Nantheera Anantrasirichai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-04323.txt.

Introduction

Paul Hill and co-authors build one system that both cleans turbulence-warped video and finds objects in it, in this 2025 paper. Heat shimmer bends and blurs surveillance footage, hurting the eye and the detector alike. Past work fixed the picture first and detected second, losing information between steps. Their framework, DMAT, trains both halves together so each teaches the other.

A three-dimensional Mamba module handles the shifting and smearing across frames. The detector's meaning-level features flow back to guide the cleaning. On test sets with synthetic turbulence, the joint system beat separate state-of-the-art cleaners and detectors by up to fifteen percent. The authors argue the lesson is general: restore for the task, not just for looks.

Catalog record

CREATORPaul Hill, Alin Achim, Dave Bull, Nantheera Anantrasirichai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-04323

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

DMAT: An End-to-End Framework for Joint Atmospheric Turbulence Mitigation and Object Detection Paul Hill, Alin Achim, Dave Bull, Nantheera Anantrasirichai Abstract Atmospheric Turbulence (AT) degrades the clarity and accuracy of surveillance imagery, posing challenges not only for visualization quality but also for object classification and scene tracking. Deep learning-based methods have been proposed to improve visual quality, but spatio-temporal distortions remain a significant issue. Although deep learning-based object detection performs well under normal conditions, it struggles to operate effectively on sequences distorted by atmospheric turbulence. In this paper, we propose a novel framework that learns to compensate for distorted features while simultaneously improving visualization and object detection. This end-to-end framework leverages and exchanges knowledge of low-level distorted features in the AT mitigator with semantic features extracted in the object detector. Specifically, in the AT mitigator a 3D Mamba-based structure is used to handle the spatio-temporal displacements and blurring caused by turbulence. Features are extracted in a pyramid manner during the mitigation stage and passed to the detector. Optimization is achieved through back-propagation in both the AT mitigator and object detector. Our proposed DMAT outperforms state-of-the-art AT mitigation and object detection systems up to a 15% improvement on datasets corrupted by generated turbulence. 1 Introduction When the temperature difference between the ground and the air increases, the thickness of each air layer diminishes, and they ascend rapidly. This leads to rapid micro-scale changes in the air’s refractive index that degrade the visual quality of video signals and in turn impacting the effectiveness of automated recognition and tracking algorithms. Objects behind the distorting layers become nearly impossible to recognize using methods designed to operate in non-distorted environments. Figure 1 : Object detection results for all object sizes and classes of our proposed method (DMAT) compared to individual AT-mitigation and object detection methods. To address the challenges posed by Atmospheric Turbulence in video processing, current learning-based approaches typically adopt a range of strategies. Some methods focus on enhancing the visual quality of videos before applying object detection to the restored outputs Mao2022single ; hill2025mamat . Others retrain existing object detectors on synthetic datasets of atmospheric turbulence Uzun2025augmenting . Another strategy includes integrating motion maps, which represent displacements caused by spatial distortions, into the network inputs of the object detectors qin2025unsupervised . Alternatively, some modify the detector’s network architectures to better handle turbulence-induced distortions Lau:ATFaceGAN:2020 ; Hu:Object:2023 . In contrast to these methods, we introduce a novel approach that aggregates information in order to enhance feature quality, effectively enabling simultaneous video restoration and object detection within a cohesive, hybrid framework. Existing methods have achieved considerable success in mitigating atmospheric turbulence; however, they often fail to ensure that restored or enhanced imagery is optimally configured for automatic object detection. A key challenge is that techniques effective in reducing wavy or ripple effects may inadvertently remove crucial features necessary for object identification within video sequences. To address this, this paper proposes a solution called DMAT ( D etection and M itigation of A tmospheric T urbulence), that not only enhances visual quality but also improves object detection accuracy, as demonstrated in Fig. 1 , where our combined architecture outperforms individual AT-mitigation and object detection methods. Our approach aims to strike a balance between video enhancement and practical applicability in surveillance systems, ensuring improvement in both domains. In this paper, we introduce a novel method that leverages 3D deformable convolutions and Structured State Space sequence (S4) models, known as Mamba. This approach addresses spatial displacement across the temporal domain, mitigates distortions such as blur, and enhances contrast. It is specifically designed to restore images affected by atmospheric turbulence and subsequently feed them into an object detector, which outputs object locations and classifications. Our joint framework is trained end-to-end with carefully designed loss functions, enabling the high-level features from the object detector and the low-level features from the turbulence mitigator to mutually enhance each other, resulting in significant improvements in both visual quality and object detection accuracy. The main contributions of this paper are summarized as follows. • A first end-to-end architecture for joint AT mitigation and object detection. We refer to this combined architecture as DMAT (Detection and Mitigation of Atmospheric Turbulence). • A new optimized set of synthetic turbulent videos based on subsets of the COCO dataset. • Benchmarking the end-to-end system (DMAT) against a range of individually applied mitigation and detection systems. 2 Related work 2.1 Atmospheric turbulence removal Atmospheric turbulence is a limiting factor in the imaging of distant objects, degrading visual quality and impairing the performance of automated object recognition and tracking systems. Image sequences captured in turbulent environments often exhibit blurring and rippling effects, accompanied by fluctuations in light intensity across the scene. This phenomenon is particularly pronounced in environments characterized by intense heat, such as deserts, heated road surfaces, and areas surrounding man-made heat sources. Moreover, atmospheric turbulence exacerbates challenges in long-distance imaging applications, such as surveillance, especially when compounded by additional factors such as fog or haze, which further degrade image clarity and quality. Other environmental parameters, including wind speed, altitude, and humidity, also contribute to the severity of these distortions. Given the spatio-temporal variability of atmospheric turbulence, restoring a scene distorted by atmospheric turbulence is a challenging problem. Traditional methods have involved frame selection, image registration, image fusion, phase alignment and image deblurring Anantrasirichai:Atmospheric:2013 ; Zhu:Removing:2013 ; Xie:Removing:2016 . Removing the turbulence distortion from a video containing moving objects is very challenging, as generally multiple frames are used and they are needed to be aligned. Temporal filtering with local weights determined from optical flow is employed to address this by Anantrasirichai et al. Anantrasirichai:Atmospheric:2018 . However, artefacts in the transition areas between foreground and background regions can remain. Removing atmospheric turbulence based on single image processing is proposed using ML by Gao et al. Gao:Atmospheric:2019 . Deep learning techniques to solve this problem are still in their early stages. However, one method reported employs a CNN to support deblurring Nieuwenhuizen:deep:2019 and another employs multiple frames using a GAN architecture Chak:Subsampled:2018 . This however appears only to work well for static scenes. Similar to de-hazing, physics-inspired models have been widely developed to remove turbulence distortion jaiswal2023physics ; Jiang_2023_CVPR , while complex-valued CNNs have been proposed to exploit phase information anantrasirichai2023atmospheric . There was also an attempt to use implicit neural representation (INR) to solve this problem, offering tile and blur correction Jiang_2023_CVPR . However, diffusion models outperform on a single image nair2023ddpm , and transformer-based methods remain state-of-the-art for restoring videos Zhang:Image:2024 ; zou2024deturb . A recent review can be found in hill2025deep . Figure 2 : Diagrams of (a) transitional approach and (b) our proposed approach for joint restoration and object detection under atmospheric turbulence. Figure 3 : Architecture of the proposed DMAT framework for atmospheric distortion mitigation and object detection. 2.2 Object Detection Object detection involves both classification and localization, determining the presence of objects and their precise positions within visual inputs. The complexity of object detection arises from challenges such as varying object scales, occlusions, and diverse backgrounds. State-of-the-art object detection methods are most often based on three core components: i) Backbone Network: A deep CNN, often pre-trained on large datasets, serves as the feature extractor. Architectures

Read the complete text