Context-Aware Semantic Segmentation via Stage-Wise Attention

Antoine Carreaud, Elias Naha, Arthur Chansel, Nina Lahellec, Jan Skaloud, Adrien Gressin · 2026

Read Context-Aware Semantic Segmentation via Stage-Wise Attention by Antoine Carreaud, Elias Naha, Arthur Chansel, Nina Lahellec, Jan Skaloud, Adrien Gressin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-11310.txt.

Introduction

Antoine Carreaud and five coauthors segment vast aerial images with care. Their 2026 preprint uses stage-wise attention for the task. Semantic segmentation labels each pixel by what it shows. Ultra-high-resolution images are key in mapping and land monitoring.

Transformer models are strong at this task but hit a wall. Memory use grows with the square of the token count. Big images force a bad choice on the model. It must shrink either context or fine spatial detail. The team's stage-wise attention works through the image in steps. That design aims to keep both wide context and fine detail. Remote sensing work needs that pair to map land well.

Catalog record

CREATORAntoine Carreaud, Elias Naha, Arthur Chansel, Nina Lahellec, Jan Skaloud, Adrien Gressin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-11310

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Context-Aware Semantic Segmentation via Stage-Wise Attention Antoine Carreaud, Elias Naha, Arthur Chansel, Nina Lahellec, Jan Skaloud, Adrien Gressin Abstract Semantic ultra high resolution image (UHR) segmentation is essential in remote sensing applications such as aerial mapping and environmental monitoring. Transformer-based models struggle in this setting because memory grows quadratically with token count, constraining either the contextual scope or the spatial resolution. We introduce CASWiT (Context-Aware Stage-Wise Transformer), a dual-branch, Swin-based architecture that injects global cues into fine-grained UHR features. A context encoder processes a downsampled neighborhood to capture long-range dependencies, while a high resolution encoder extracts detailed features from UHR patches. A cross-scale fusion module, combining cross-attention and gated feature injection, enriches high-resolution tokens with context. Beyond architecture, we propose a SimMIM-style pretraining. We mask 75% of the high-resolution image tokens and the low-resolution center region that spatially corresponds to the UHR patch, then train the shared dual-encoder with small decoder to reconstruct the UHR initial image. Extensive experiments on the large-scale IGN FLAIR-HUB aerial dataset demonstrate the effectiveness of CASWiT. Our method achieves 65.83% mIoU, outperforming RGB baselines by 1.78 points. On URUR, CASWiT achieves 49.1% mIoU, surpassing the current SoTA by +0.9% under the official evaluation protocol. All codes are provided on: https://huggingface.co/collections/heig-vd-geo/caswit . 1 Introduction Semantic segmentation of remote sensing imagery is fundamental to many key geospatial applications, including land-use mapping, environmental monitoring, and disaster response. As these applications become more reliant on ultra-high-resolution (UHR) aerial imagery for capturing fine spatial details, it is crucial that the methods employed preserve local structures while also considering broader spatial contexts. Transformer-based architectures 10 ; 29 ; 42 ; 7 have advanced state-of-the-art results in vision tasks. However, their application to ultra-high-resolution (UHR) inputs is limited by their quadratic complexity and GPU memory constraints. Common workarounds, such as downsampling or tiling, either truncate the context necessary for interpreting complex scenes, or result in a loss of high resolution, impairing segmentation at both the object and scene levels. Recent UHR approaches therefore combine high-resolution patch processing with explicit context modeling via multi-branch or cross-scale designs 2 ; 34 ; 21 , emphasizing the importance of global context and the consequences of its loss. Our approach. CASWiT , a dual-branch hierarchical transformer, is introduced for RGB-only UHR segmentation. One branch processes high-resolution (HR) crops in order to preserve high resolution, boundaries and small objects, while a second branch ingests wider low-resolution (LR) patches in order to encode global context. These two streams interact at multiple encoder stages through compact global cross-attention blocks (HR queries over LR keys/values), enabling early context injection while remaining compute-efficient. This approach introduces context early on while keeping the computation efficient. To further improve cross-scale learning, masked image modeling (SimMIM-style 46 ) is adapted to the dual-stream setting and pre-trained using large amounts of unlabelled orthophotos. Benchmarks. The primary evaluation is conducted on FLAIR-HUB 12 using an RGB-only UHR protocol that leverages its geospatial structure: although FLAIR-HUB was not originally designed for UHR segmentation (it targets multi-modal/time-series Earth Observation), its georeferenced orthophotos can be reassembled into large, contiguous tiles that preserve long-range spatial context and enable UHR evaluation. Compared with URUR 20 , FLAIR-HUB offers a substantially larger scale and more carefully curated annotations (see § 4.1 ), making it a more effective testbed for cross-scale learning. For continuity with prior work, we also report results on URUR . Finally, while multi-modal inputs can boost performance on FLAIR-HUB 12 ; 23 , we emphasize that RGB-only UHR segmentation remains a fundamental and practically relevant challenge in Earth Observation. Contributions. • We introduce CASWiT (Context-Aware Stage-Wise Transformer), a dual-branch architecture that performs stage-wise cross-attention from HR to LR features, allowing context to be injected early in the encoding hierarchy, while preserving fine-grained details from the HR branch. • We design a dual-stream SimMIM pretraining strategy that strengthens cross-scale learning and demonstrates effective transfer to large-scale UHR segmentation tasks. • We establish an RGB-only UHR evaluation protocol on FLAIR-HUB , exploiting its geospatial structure to reconstruct large contiguous tiles. CASWiT consistently improves over prior RGB-only state-of-the-arts on FLAIR-HUB-RGB and URUR; all code, pretrained weights, and evaluation scripts will be released for reproducibility. 2 Related Work Dual-stream UHR segmentation. Processing ultra-high-resolution (UHR) imagery for semantic segmentation typically requires preserving fine details while aggregating long-range context. A widely adopted strategy is to treat UHR segmentation as a dual-stream fusion problem: an HR stream for local structures and a complementary LR/context stream for scene-level semantics. GLNet 5 popularized this formulation with CNN backbones and late fusion by concatenation. Subsequent methods refine this template along architecture and fusion axes. WSDNet 20 integrates multi-level discrete wavelet transforms (DWT) into the UHR stream and introduces a Wavelet Smooth Loss to preserve structured context and fine textures while reducing computation, rather than relying on a single downsampled global stream with late fusion. FCtL 33 fuses features extracted at three cropping scales (CNN-based) via mid-level integration. GPWFormer 21 mixes CNNs and transformers in a dual-branch design with late fusion, while SGNet 40 introduces attention-based feature extractors in both branches prior to concatenation. STUNet 18 embeds a Transformer (specifically the Swin Transformer) and a CNN in parallel encoders and then uses a relational aggregation module to integrate global and local features hierarchically. More recently, DESformer 26 further explores dual-encoder architectures combining transformer and CNN branches for enhanced feature interaction. Beyond these, several families target complementary aspects of UHR efficiency and accuracy. (i) Iterative/local patching with global guidance: GLNet’s original strategy inspired follow-ups that iteratively process UHR patches under LR guidance. (ii) Proposal/selection to reduce UHR cost: PPN 44 ; 22 allocates compute to informative regions only. (iii) Shallow all-pixel models: ISDNet 14 , WSDNet 20 , GPWFormer 21 , and SGHRQ 24 aim to process full UHR images with reduced complexity to avoid iterative patching, often correcting HR predictions using LR-derived context. Recent memory-efficient approaches like WCTNet 4 , EFFNet 36 , and RingFormer-Seg 48 tackle the memory-accuracy tradeoff through token filtering, global-local branches, and efficient fusion strategies. (iv) Boundary refinement/ensemble: CascadePSP 8 , MagNet 19 , and FCtL 33 emphasize progressive refinement and multi-scale ensembling to fix boundary errors. Most of these approaches rely on mid/late fusion, where HR and LR features interact after substantial single-stream processing, whereas early fusion can be beneficial when both inputs share similar representational spaces 1 . Single-stream HR backbones. An alternative to dual-stream fusion is to rely on hierarchical vision transformers or multi-scale CNNs that capture locality and globality within a single stream. Representative examples include Swin Transformer 29 ; 28 and PVT 41 ; 42 , alongside UHR-oriented single-stream variants 43 ; 15 ; 47 ; 27 ; 37 ; 35 .While these models improve scalability compared to vanilla ViT, balancing spatial resolution and memory for truly UHR inputs remains challenging. Fusion mechanisms and module placement. Despite the intuitive complementarity of HR and LR signals, many dual-stream models still perform simple mid/late fusion (concatenation/summation). Attention-based alignment between heterogeneous resolutions is less explored at multiple depths, even though cross-attention provides a systematic mechanism to condition HR features on LR context (and vice versa) early enough to guide subsequent hierarchy building. Works like DESformer 26 introduces a mid-level, multi-depth Feature Interaction Module that

Read the complete text