Multi-Modal Building Inspection via Perceiver IO Fusion of Satellite and Street-Level Imagery

Niels Sombekke, Rob G.J. Wijnhoven, Martin R. Oswald · 2026

Read Multi-Modal Building Inspection via Perceiver IO Fusion of Satellite and Street-Level Imagery by Niels Sombekke, Rob G.J. Wijnhoven, Martin R. Oswald (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26381.txt.

Introduction

Niels Sombekke, Rob Wijnhoven, and Martin Oswald fuse satellite and street-level imagery with Perceiver IO on patch tokens from one shared DINOv2 backbone. The design handles street-level views for each building without padding or fixed pooling, predicting roof element and material classes. Their dataset spans over thirty-two thousand buildings in ten countries, joining satellite images to as many as eight street views per segment. RGB-M masking, which appends the footprint mask as a fourth channel, beats hard cropping, and Perceiver IO beats rivals, in a 2026 paper.

Catalog record

CREATORNiels Sombekke, Rob G.J. Wijnhoven, Martin R. Oswald
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34080
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26381

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multi-Modal Building Inspection via Perceiver IO Fusion of Satellite and Street-Level Imagery Niels Sombekke, Rob G.J. Wijnhoven, Martin R. Oswald Abstract We present a multi-modal classification framework that fuses satellite and street-level imagery through a Perceiver IO architecture operating on spatial patch tokens from a shared DINOv2 backbone. The design naturally handles a variable number of street-level views per building without padding or fixed-size pooling, and jointly predicts multi-label roof element and roof material classes. We construct a large-scale dataset of 32,135 buildings (61,672 segments) spanning ten countries, pairing satellite images with up to eight street-level views per segment and evaluating four masking strategies for isolating the target building. We propose an RGB-M masking strategy that appends the building footprint mask as a fourth input channel, providing a soft spatial prior that outperforms hard cropping across both modalities. The Perceiver IO fusion model improves over all other fusion strategies and yields substantial per-class gains for attributes visible from street level (e.g., +11.3 AP for slate, +1.3 AP for dormers), though the satellite-only baseline retains a slight advantage in macro-averaged mAP for classes that are predominantly visible from above. These results establish a scalable, flexible architecture for multi-modal building inspection that can accommodate heterogeneous inputs and multiple output tasks. 1 Introduction Figure 1 : Example input data for a building: satellite image (left) and corresponding street-level images (right). Satellite imagery captures roof geometry and context, while street-level views reveal facade details, materials, and structural elements that are invisible from above. Maintaining and insuring the built environment at national scale requires detailed knowledge of individual buildings: what materials cover the roof, which structural elements are present, and what condition they are in. Traditional inspection relies on labor-intensive site visits that are costly, slow, and prone to human error, while generating significant CO 2 emissions from repeated travel 27 . With the European building stock alone comprising over 200 million structures, 40% of the EU’s energy consumption and 36% of its greenhouse gas emissions 5 , manual approaches simply cannot keep pace with the demand for up-to-date property information. A scalable, data-driven alternative is needed. Remote sensing offers a scalable path. Satellite and aerial imagery can be collected over large areas, updated regularly, and processed algorithmically, enabling the analysis of thousands of properties without physical visits. Satellite data captures a building’s footprint, roof geometry, and surrounding context from above, while street-level imagery—increasingly available through services such as Google Street View—provides close-range views of facades, wall conditions, and material textures that are invisible from a top-down perspective. Combining both modalities can produce richer, multi-perspective building profiles than either source alone 2 ; 11 ; 22 . However, fusing satellite and street-level imagery poses several challenges. First, the data itself is heterogeneous: satellite images may have limited resolution and suffer from misaligned building footprints, while street-level views are frequently occluded by vegetation, vehicles, or neighboring structures 10 . Second, an effective method must isolate the target building from its surroundings—leveraging the building footprint to focus on the relevant structure—while retaining enough environmental context to support classification. Third, the number of available street-level images varies widely across buildings: some have dozens of views from different angles, others have none at all. A practical architecture must therefore handle a variable number of inputs without degrading performance on buildings with sparse or missing street-level coverage. We address these challenges with a Perceiver IO-based fusion architecture 15 that operates directly on spatial patch tokens extracted by a shared DINOv2 26 backbone. Unlike feature-vector fusion approaches that discard spatial structure through global pooling, our model preserves fine-grained patch-level information from both modalities and compresses them through cross-attention into a fixed-size latent representation. The architecture naturally accommodates varying numbers of street-level inputs through its set-based cross-attention mechanism, and jointly predicts multi-label roof element and roof material classes via task-specific output heads. We make the following contributions: 1. We develop a Perceiver IO fusion model that processes spatial tokens from satellite and street-level imagery, outperforming concatenation and transformer-based fusion baselines and delivering substantial gains for classes where street-level views are informative. 2. We systematically evaluate four masking strategies for isolating the target building and show that an RGB-M approach, appending the binary footprint mask as a fourth channel, provides the best trade-off between spatial focus and contextual information. 3. We construct a multi-modal building inspection dataset of 32,135 buildings (61,672 segments) across ten countries, pairing satellite imagery with variable numbers of street-level views and building footprint masks. 4. We compare two backbones (ResNet-50, DINOv2-S) and multiple fine-tuning strategies, demonstrating that fully fine-tuned DINOv2-S consistently outperforms the convolutional baseline on both modalities. 2 Related Work Satellite imagery for building analysis. CNN-based methods have been widely applied to satellite imagery for building footprint extraction 16 ; 38 ; 17 , damage detection 36 , and roof type identification 19 . Vision Transformers have since improved large-scale building extraction 33 and damage assessment 18 through their ability to capture global spatial context. More recently, self-supervised foundation models such as DINOv2 26 have demonstrated strong transfer to remote sensing tasks, particularly in low-label regimes 24 ; 28 . Despite these advances, relying solely on satellite imagery limits inspection to attributes visible from above, missing facade-level detail that is critical for comprehensive building assessment. Street-level imagery for building analysis. Street-level images offer ground-level perspectives on facades, materials, and structural elements that are occluded in satellite views. Deep learning approaches have been applied to building typology classification 7 ; 31 ; 21 , property value estimation 37 , facade parsing 20 ; 32 , and floor count estimation 13 . Semi-supervised approaches combine street-level imagery with auxiliary data such as OpenStreetMap metadata for building height estimation 23 . Biljecki and Ito 1 provide a comprehensive survey of street-view imagery in urban analytics. Key challenges include variable lighting and weather conditions, occlusion by vegetation and vehicles, inconsistent camera calibration across regions, and privacy-related blurring 10 . Fusion of satellite and street-level modalities. Combining satellite and street-level data leverages their complementary strengths: broad spatial coverage from above and detailed facade information from ground level. This multi-modal approach has improved performance in house price estimation 22 , land use classification 2 ; 29 , building type classification 11 , socioeconomic prediction 30 , and flood risk assessment 35 . Early fusion methods concatenate or sum global feature vectors, treating modalities symmetrically without exploiting their complementary spatial structure 6 . Recent attention-based approaches process multiple views with learned weighting, focusing on the most informative inputs 3 ; 12 ; 8 . However, most existing methods operate on a fixed number of views per building, discarding spatial token-level detail through global pooling before fusion. Our work differs in three respects. First, we fuse at the spatial token level rather than the feature vector level, preserving fine-grained patch information from both modalities. Second, we employ Perceiver IO 15 to compress a variable-length token sequence into a fixed-size latent representation through cross-attention, naturally handling buildings with any number of street-level views. Third, we build on a self-supervised DINOv2 backbone rather than task-specific supervised features, providing richer representations for both modalities. 3 Method We describe the dataset construction ( Section 3.1 ), masking strategies for isolating

Read the complete text