LALE: Lightweight-Transformer Architecture for Land-Cover Estimation
Ümit Mert Çağlar, Alptekin Temizel · 2026
Read LALE: Lightweight-Transformer Architecture for Land-Cover Estimation by Ümit Mert Çağlar, Alptekin Temizel (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-02092.txt.
Introduction
Çağlar and Temizel present a light transformer for land-cover mapping from above. Semantic segmentation of remote sensing images is the task. Models must capture global context and local detail together. Tight compute budgets constrain field use of such models. Past work optimizes one axis at the cost of others.
Their LALE architecture splits its encoder by resolution. Light ConvMixer stages handle high-resolution local features. Transformer stages handle low-resolution global context. Self-attention's quadratic cost is confined to deep, small maps. An all-MLP decoder merges features across scales. The design is end to end for land-cover estimation. It aims for accuracy within strict compute limits. Hybrid designs without heavy backbones are the target.
Catalog record
| CREATOR | Ümit Mert Çağlar, Alptekin Temizel |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 27 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.02092
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LALE: Lightweight-Transformer Architecture for Land-Cover Estimation Ümit Mert Çağlar, Alptekin Temizel Abstract Semantic segmentation of remote sensing imagery requires models that capture both global context and local detail under tight computational budgets. Prior work typically optimizes for one of these axes: attention for global context, convolution for local detail, or compactness for efficiency. While hybrid approaches aim to capture both, they require architectural changes and encoder backbones with computational overhead, limiting efficiency and performance. We present LALE (Lightweight-transformer Architecture for Land-cover Estimation), an end-to-end remote sensing image segmentation architecture, that bifurcates its encoder by resolution: lightweight ConvMixer stages handle high-resolution local features, while transformer stages handle low-resolution global context, confining the quadratic cost of self-attention to deep, downsampled feature maps. An all-MLP multi-scale decoder, together with RMSNorm and StarReLU throughout, further reduces compute and parameter count. On the large-scale ARAS400k remote-sensing segmentation benchmark, LALE establishes a strong efficiency-performance trade-off against CNN, transformer, and hybrid baselines. Our smallest variant, (just 1.6M parameters), reaches within 2.6 F1 points of the best baseline (UPerNet) while using 4.5 × 4.5\times fewer parameters, 7 × 7\times less storage, 17 × 17\times fewer GMACs, and delivering 1.8 × 1.8\times higher throughput. The codebase for LALE is publicly available at https://github.com/caglarmert/LALE . keywords Transformer Models, Deep Convolutional Networks, Image Segmentation, Remote Sensing † † authorinfo: Further author information: (Send correspondence to Ümit Mert Çağlar) Ümit Mert Çağlar: E-mail: [email protected] Alptekin Temizel: E-mail: [email protected] 1 Introduction Semantic segmentation of remote sensing images is crucial for a wide range of applications, including environmental monitoring, urban planning, disaster awareness and precise land-cover estimation. Convolutional Neural Networks (CNNs) bring strong inductive biases and computational efficiency, but their limited receptive fields restrict long-range reasoning. Vision transformers model global context effectively through self-attention, yet their quadratic cost in spatial resolution makes them expensive. This is a particularly important drawback in remote sensing, where inputs are high-resolution, throughput requirements are large, and deployment platforms are often resource-constrained. Hybrid architectures aim to combine the strengths of both families, but most are assembled from generic ImageNet backbones paired with parameter-heavy decoders, inheriting overhead that is not justified by the demands of the segmentation task itself. As a result, the efficiency-performance trade-off in remote sensing segmentation remains an open problem. In this work, we introduce Lightweight-Transformer Architecture for Land-Cover Estimation (LALE), an end-to-end segmentation model designed specifically for this trade-off, illustrated in Figure 1 . LALE bifurcates its encoder by resolution: lightweight ConvMixer stages process high-resolution feature maps where local detail dominates, while transformer stages operate on deeper, downsampled feature maps where global context matters and the quadratic cost of self-attention is tractable. A lightweight all-MLP multi-scale decoder replaces conventional heavy upsampling heads, and operation-level choices (RMSNorm in place of LayerNorm, StarReLU in place of GELU, and 3\times 3 stride-2 convolutions in the stem) further reduce compute without degrading representational capacity. Extensive experiments on the large-scale ARAS400k benchmark show that LALE achieves a strong performance-efficiency trade-off. Against CNN baselines, LALE matches segmentation accuracy at a fraction of the parameter and compute budget; against dense-prediction transformer baselines, it delivers comparable accuracy with one to two orders of magnitude lower compute and substantially higher throughput, making it well-suited to real-time and resource-constrained deployment. Our contributions are: • We introduce LALE, a lightweight attention-convolution hybrid hierarchical architecture for semantic segmentation of remote sensing images, which optimize the computational efficiency and segmentation task performance. • LALE, a resolution-bifurcated convolution-transformer hybrid encoder paired with an all-MLP multi-scale decoder, designed end-to-end for remote sensing semantic segmentation. • A set of operation-level efficiency choices (RMSNorm, StarReLU, and small-kernel strided downsampling) that together reduce compute and memory footprint without sacrificing accuracy. • A comprehensive benchmark on ARAS400k spanning CNN, transformer, and hybrid architectures, establishing an extended reference point for future work on this dataset. • Empirical evidence that LALE occupies a favorable region of the accuracy-efficiency Pareto frontier. Figure 1: Overall approach for the Lightweight-Transformer Architecture for Land-cover Estimation (LALE) 2 Related Work Semantic segmentation in remote sensing presents unique challenges, including extreme scale variations, high-resolution processing demands, and complex, heterogeneous backgrounds. To address these, early architectural paradigms relied heavily on CNNs, with UNet [ 16 ] establishing a robust baseline through a symmetrical encoder-decoder structure and lateral skip connections to recover spatial hierarchies. Expanding on this foundation to better handle dynamic multi-scale fusion, UNet++ [ 26 ] introduced densely connected, cross-fused skip connections spanning varying scaling ratios, enabling the hyper-fine segmentation of complex objects. Parallel structural adaptations, such as Linknet [ 4 ] , utilized lightweight encoders and additive skip connections to streamline memory utilization and accelerate spatial processing. The Feature Pyramid Network (FPN) [ 12 ] emerged to inject high-level abstract semantics across multiple spatial resolutions via a bidirectional structural pathway. Pyramid Attention Network (PAN) [ 9 ] merged CNNs with spatial attention mechanisms to fuse multi-scale context and provide precise pixel-level attention for small targets without relying on dilated convolutions. Concurrently, the DeepLabV3 [ 5 ] series advanced multi-scale context capture using Atrous Spatial Pyramid Pooling (ASPP), with DeepLabV3+ [ 6 ] integrating a dedicated decoder and depthwise separable convolutions to recover remarkably sharp architectural boundaries. The Unified Perceptual Parsing Network (UPerNet) [ 22 ] coupled an FPN with a Pyramid Pooling Module (PPM) to learn deep global pairwise relationships, actively mapping intra-class consistencies across expansive land-cover zones. This continuous drive toward broader spatial understanding ultimately set the stage for attention-driven architectures like SegFormer [ 23 ] , which adopted a positional-encoding-free Mix Transformer (MiT) encoder and a lightweight MLP decoder, granting the model an effectively infinite receptive field to capture long-range environmental dependencies. Recent advancements in vision transformers (ViTs) have focused on enhancing the training stability and scalability of base architectures. DeiT3 [ 18 ] demonstrates that by refining training recipes, incorporating advanced data augmentation and regularization, standard ViTs can significantly outperform previous iterations without increasing model complexity. Similarly, SwinV2 [ 13 ] introduces a post-normalization configuration and log-spaced relative position biases, achieving better stability and effectively adapts to large-scale visual recognition tasks. To bridge the gap between local and global context, several models utilize hybrid attention mechanisms such as MaxViT [ 20 ] , introducing Multi-Axis Attention that decomposes the standard attention mechanism into local window attention and global sparse grid attention. This allows for global-local spatial interactions with linear complexity relative to the image size, making it highly versatile across different scales. A significant trend in transformer research is the optimization for real-time inference on edge devices. EfficientFormer [ 11 ] re-evaluates the design bottlenecks of ViTs, proposing a dimension-consistent design that integrates the speed of Convolutional Neural Networks (CNNs) with the performance of transformers. FastViT [ 21 ] builds upon this by utilizing structural reparameterization and a novel RepMixer block. It minimizes latency by reducing memory access costs, making it one of the most efficient hybrid models for mobile deployment. 3 Method We introduce a lightweight attention-convolution hybrid hierarchical architecture (LALE) for remote sensing semantic segmentation (Figure 2 ). LALE integrates global context modeling capabilities of attention-based architectures (transformers) and the local inductive biases of CNNs. The architecture contains three distinct modules, an overlapping patch embedding stem, a four-stage bifurcated hybrid encoder and a lightweight multi-scale decoder. Figure