Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection
Irem Ulku, Erdem Akagündüz, Ömer Özgür Tanrıöver · 2026
Read Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection by Irem Ulku, Erdem Akagündüz, Ömer Özgür Tanrıöver (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15856.txt.
Introduction
Ulku, Akagündüz, and Tanrıöver present CBC-SLP, a model for multispectral semantic segmentation that copes when some sensor inputs are missing. Standard shared-representation models can lose modality-specific detail and even slip when all inputs are present. Their structured latent projection splits latent features into shared and modality-specific parts, then routes them to the decoder based on which modalities are available. Tests on three remote sensing image sets show it beats leading models in both full and missing-modality cases.
Catalog record
| CREATOR | Irem Ulku, Erdem Akagündüz, Ömer Özgür Tanrıöver |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 74798 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.15856
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection Irem Ulku, Erdem Akagündüz, Ömer Özgür Tanrıöver Abstract Multimodal remote sensing data provide complementary information for semantic segmentation, but in real-world deployments, some modalities may be unavailable due to sensor failures, acquisition issues, or challenging atmospheric conditions. Existing multimodal segmentation models typically address missing modalities by learning a shared representation across inputs. However, this approach can introduce a trade-off by compromising modality-specific complementary information and reducing performance when all modalities are available. In this paper, we tackle this limitation with CBC-SLP, a multimodal semantic segmentation model designed to preserve both modality-invariant and modality-specific information. Inspired by the theoretical results on modality alignment, which state that perfectly aligned multimodal representations can lead to sub-optimal performance in downstream prediction tasks, we propose a novel structured latent projection approach as an architectural inductive bias. Rather than enforcing this strategy through a loss term, we incorporate it directly into the architecture. In particular, to use the complementary information effectively while maintaining robustness under random modality dropout, we structure the latent representations into shared and modality-specific components and adaptively transfer them to the decoder according to the random modality availability mask. Extensive experiments on three multimodal remote sensing image sets demonstrate that CBC-SLP consistently outperforms state-of-the-art multimodal models across full and missing modality scenarios. Besides, we empirically demonstrate that the proposed strategy can recover the complementary information that may not be preserved in a shared representation. The code is available at https://github.com/iremulku/Multispectral-Semantic-Segmentation-via-Structured-Latent-Projection-CBC-SLP- . I Introduction Accurate land-cover mapping is vital for Earth observation applications. In particular, remote sensing semantic segmentation [ 29 , 3 ] , which enables large-scale mapping of crops, trees, and forests, supports agricultural monitoring [ 39 ] and ecological analysis [ 18 ] . However, relying on a single remote sensing modality limits segmentation performance, since complementary modalities provide mutually reinforcing information for class separation [ 26 , 19 ] . Multispectral optical data, including RGB, near-infrared (NIR), and short-wave infrared (SWIR) bands, enhance class discrimination by providing spectral cues sensitive to chlorophyll and moisture [ 27 ] . Height and terrain information from digital surface models (DSM) and digital elevation models (DEM) improves separability by providing geometric context that complements spectral information [ 47 ] . To illustrate the benefit of multimodal fusion, Fig. 1 shows an example from Hunan, China, where the Sentinel-2 multispectral optical image is paired with a DEM height map over the same area to highlight complementary spectral and topographic cues for land-cover mapping [ 17 ] . Moreover, synthetic aperture radar (SAR) offers structure-sensitive information that enhances robustness, particularly when optical imagery is degraded under challenging illumination or atmospheric conditions [ 34 ] . Therefore, integrating these complementary modalities is promising for achieving accurate semantic segmentation across diverse scenes [ 12 , 35 ] . Fig. 1 : Multispectral optical (false RGB: infrared-red-green channels) image on the left and DEM height map on the right, both from the Hunan image set [ 17 ] . While the multispectral image highlights surface reflectance and land-cover differences, the DEM height map reveals the terrain structure, providing complementary information for scene interpretation. Although fusing remote sensing data with different spatial, spectral, and angular resolutions is effective [ 11 ] , some modalities are not always available, e.g., due to varying electro-optical configurations, malfunctions in satellite sensors or challenging atmospheric conditions [ 42 , 33 , 6 ] . For instance, a hardware failure in the onboard imaging instrument of the Landsat-7 satellite platform may lead to systematic data gaps in the images [ 21 ] . Similarly, due to the dense cloud cover during the acquisition of MODIS sensor observations, the resulting satellite images exhibit information loss [ 38 ] . SAR data collection is also challenging because certain surface conditions reduce coherence, and processing errors often yield low raw data quality [ 13 ] . These challenges lead to the failure of existing fusion models, as these approaches generally rely on the complete availability of all modalities during the training and inference phases [ 48 , 4 ] . While current approaches often work well on the modality sets used during training, their performance declines on untrained modalities at test time [ 25 ] . Although training separate networks for each possible modality combination is conceptually straightforward, this strategy becomes computationally ineffective when the number of input modalities is large. Recent multimodal semantic segmentation approaches for remote sensing under incomplete modality settings aggregate complementary cross-modal information and compensate for the missing modality through transfer, reconstruction, or relation propagation. For instance, MetaRS [ 48 ] jointly learns modality-specific representations and a shared meta-modal representation across all modalities in the EarthMiss benchmark. Then, the missing information is recovered via transfer in this meta-modal space. RobSense proposes a single foundation model in which missing inputs are restored by a unimodal latent reconstructor when spectral bands or temporal observations are absent [ 9 ] . The multimodal transformer in [ 4 ] integrates heterogeneous modalities into a unified representation using learnable fusion tokens and modality attention. This approach is capable of handling missing modalities both under training and inference phases via masked self-attention. MHHL [ 12 ] models cross-modal relations through a heterogeneous hypergraph and employs information propagation to compensate for missing modalities. Primarily, these approaches focus on improving performance under missing-modality scenarios and rarely address the trade-off between missing modality robustness and optimal accuracy when all modalities are available. However, we posit that the nature of the challenge lies in achieving a unified fusion mechanism that is robust to missing modalities without sacrificing performance when all modalities are available. Given this observation, we aim to develop a unified semantic segmentation model that operates effectively in both cases. Empirical results have shown that exact modality alignment is generally sub-optimal for downstream prediction tasks; instead, constructing meaningful latent modality structures that preserve both shared and modality-specific information can improve model performance [ 15 ] . Our key insight is that by projecting a multimodal representation into shared and modality-specific subspaces, the model may retain useful information under missing modality conditions while also improving representational capacity when all modalities are available. Specifically, we introduce a novel structure that splits the latent multimodal representation into shared and modality-specific components, improving robustness under random modality dropout while mitigating the performance drop in the full-modality setting. Recently, [ 16 ] introduced a framework that targets strong performance under both full and missing modality settings by explicitly modeling missing modality conditions through additional loss terms. In contrast to such loss-based designs, we split and recombine the shared and modality-specific representations at the architectural level throughout the pipeline, rather than relying on extra loss terms. Since explicit loss-based alignment does not always guarantee meaningful latent modality structures [ 46 ] , we aim to introduce inductive bias through structured latent space projection without relying on additional adaptations of loss functions. Therefore, instead of compressing correlation-aware features into a single latent space as in [ 28 ] , we adopt a two-part decomposition that isolates shared correlation and preserves modality-specific information. Projecting the latent representation into shared and modality-specific components reduces gradient interference between full and