HiRes-FusedMIM: A High-Resolution RGB- digital surface model (DSM) Pre-trained Model for Building-Level Remote Sensing Applications Journal: ISPRS Journal

Guneet Mutreja, Philipp Schuegraf, Ksenia Bittner · 2025

Read HiRes-FusedMIM: A High-Resolution RGB- digital surface model (DSM) Pre-trained Model for Building-Level Remote Sensing Applications Journal: ISPRS Journal by Guneet Mutreja, Philipp Schuegraf, Ksenia Bittner (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-18540.txt.

Introduction

City maps from air photos gain depth when height data joins the color view. Guneet Mutreja, Philipp Schuegraf, and Ksenia Bittner train a base model on both, in work from twenty twenty-five. Past base models skip height maps, which hold key facts for building-level work. Their model uses two encoders, one per view, trained by hiding patches and rebuilding them.

A twin loss mixes rebuild and contrast goals. Tests on sorting, zone painting, and single-building cuts beat past geo methods on key sets. Height in training always helped over color alone. Split encoders beat fused ones as well. The team aims the model at digital twin work, where each building counts on its own.

Catalog record

CREATORGuneet Mutreja, Philipp Schuegraf, Ksenia Bittner
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-18540

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

HiRes-FusedMIM: A High-Resolution RGB- digital surface model (DSM) Pre-trained Model for Building-Level Remote Sensing Applications Journal: ISPRS Journal Guneet Mutreja, Philipp Schuegraf, Ksenia Bittner Abstract Recent advances in self-supervised learning have led to the development of foundation models that have significantly advanced performance in various computer vision tasks. However, despite their potential, these models often overlook the crucial role of high-resolution digital surface models (DSMs) in understanding urban environments, particularly for building-level analysis, which is essential for applications like digital twins. To address this gap, we introduce HiRes-FusedMIM, a novel pre-trained model specifically designed to leverage the rich information contained within high-resolution RGB and DSM data. HiRes-FusedMIM utilizes a dual-encoder simple masked image modeling (SimMIM) architecture with a multi-objective loss function that combines reconstruction and contrastive objectives, enabling it to learn powerful, joint representations from both modalities. We conducted a comprehensive evaluation of HiRes-FusedMIM on a diverse set of downstream tasks, including classification, semantic segmentation, and instance segmentation. Our results demonstrate that: 1) HiRes-FusedMIM outperforms previous state-of-the-art geospatial methods on several building-related datasets, including WHU Aerial and LoveDA, demonstrating its effectiveness in capturing and leveraging fine-grained building information; 2) Incorporating DSMs during pre-training consistently improves performance compared to using RGB data alone, highlighting the value of elevation information for building-level analysis; 3) The dual-encoder architecture of HiRes-FusedMIM, with separate encoders for RGB and DSM data, significantly outperforms a single-encoder model on the Vaihingen segmentation task, indicating the benefits of learning specialized representations for each modality. To facilitate further research and applications in this direction, we will publicly release the trained model weights. PACS 0000 , 1111 MSC 0000 , 1111 † † highlights: A unique dataset of over 368k paired RGB- DSM images at 0.2-0.5 m resolution is curated from various sources across Europe to train a multimodal pre-trained model- HiRes-FusedMIM. A novel dual encoder SimMIM architecture facilitating joint representation learning is developed. Demonstrated the effectiveness of HiRes-FusedMIM on a diverse set of building-related tasks, including classification, semantic segmentation, and instance segmentation. 1 Introduction Accurate and up-to-date information about buildings is crucial for a wide range of applications, from urban planning and disaster response to the creation of digital twins. Detailed 3D building models, in particular, are essential for simulating urban environments, assessing risks, and optimizing infrastructure management. High-resolution imagery, particularly the fusion of RGB and DSMs , plays a crucial role in extracting detailed building information. While RGB data captures spectral characteristics of the surface of the Earth, DSMs provide critical elevation information, enabling the derivation of building heights and 3D structures. In recent years, foundation models, pre-trained on massive datasets using self-supervised learning, have revolutionized computer vision and shown great promise in remote sensing [ 8 , 35 , 6 , 21 ] . These models learn general-purpose visual representations that can be adapted to various downstream tasks, including those requiring precise building information. Despite the advantages of foundation models and the importance of high-resolution DSMs , existing approaches in remote sensing often fall short in fully leveraging the potential of this combination. This gap stems primarily from the limited availability of large-scale, high-resolution RGB- DSM datasets and the inherent challenges in effectively fusing these modalities during pre-training. Models predominantly trained on open-source data like Sentinel-2 imagery ( 10\text{\,}\mathrm{m} resolution) [ 21 ] often struggle to generalize well to tasks requiring finer details [ 34 ] . While some efforts have incorporated high-resolution RGB data [ 26 ] , the development of pre-trained models that effectively leverage high-resolution DSMs , especially for building-level analysis, remains a significant area for explorations. This paper addresses this gap by introducing HiRes-FusedMIM , a novel pre-trained model specifically designed for building-level remote sensing applications. HiRes-FusedMIM leverages a unique and extensive dataset of paired RGB- DSM imagery at 0.2\text{\,} - 0.5\text{\,}\mathrm{m} resolution, carefully curated from various sources. The model utilizes a dual encoder simple masked image modeling (SimMIM) architecture [ 40 ] with a multi-objective loss function that combines both reconstruction and contrastive objectives. We demonstrate the effectiveness of this approach on a diverse range of downstream tasks, including classification, semantic segmentation, and instance segmentation, showing its superior performance in capturing and leveraging high-resolution, multi-modal information for detailed building analysis. The paper is organized as follows: Section 2 reviews related work on importance of DSMs , self-supervised learning, and foundation models in remote sensing. Section 3 details the HiRes-FusedMIM methodology, including the dataset, model architecture, and pre-training process. Section 4 presents the experimental results, comparing our approach to relevant baselines and state-of-the-art methods. Section 5 presents a series of ablation studies analyzing the impact of different model components. Finally, Section 6 discusses the implications of our findings and outlines future research directions. 2 Related work This section provides a comprehensive review of existing research relevant to the development and applications of HiRes-FusedMIM. We first discuss the significance of DSMs in remote sensing, followed by a discussion of recent advancements in self-supervised learning and foundation models for remote sensing. Finally, we highlight the motivation and contributions of our research, emphasizing the novelty of our approach. 2.1 The Importance of DSMs in Remote Sensing DSMs are essential data sources in remote sensing, providing crucial elevation information about the surface of the Earth and all objects upon it. Unlike digital terrain models (DTMs), which represent the bare earth, DSMs capture the heights of buildings, vegetation, trees, and other structures. This detailed elevation data is essential for a wide range of applications, including: 1. Urban Planning and Analysis: DSMs are used to analyze urban morphology, model urban growth, assess urban density, and aid in infrastructure planning and management. 2. Forestry and Vegetation Management: DSMs are used to estimate forest canopy heights, analyze forest structure, monitor deforestation, and assess biomass. 3. Disaster Management: DSMs play a vital role in flood modeling, landslide risk assessment, and damage assessment after natural disasters. 4. 3D Building Reconstruction: DSMs provide the height information necessary to create realistic 3D building models, which are essential for applications like urban planning, virtual city modeling, and digital twins. The recent advancements in deep learning have significantly improved the utilization of DSMs in remote sensing, enabling the development of more automated and accurate methods for various tasks. Deep learning models, particularly convolutional neural networks (CNNs) , have demonstrated success in: 1. Semantic Segmentation of DSMs : CNNs can effectively classify DSM pixels into different categories, such as buildings, vegetation, and water bodies, enabling the automatic extraction of features with high accuracy [ 4 , 5 , 30 , 33 ] . 2. Joint Learning from RGB and DSM Data: By combining RGB and DSM data, deep learning models can learn richer representations, leveraging both spectral and elevation information to achieve superior performance in tasks like land cover classification, object detection, and semantic segmentation [ 5 , 14 , 32 , 31 ] . 3. Generating levels of detail (LoD) 2-Level Building Models: Recent studies have demonstrated the potential of deep learning in achieving LoD 2-level 3D building reconstruction by extracting roof planes and other detailed features from DSMs [ 14 , 41 , 31 , 15 ] . This progress in deep learning, combined with the increasing availability of high-resolution DSM data, has opened new possibilities for

Read the complete text