ZAYAN: Disentangled Contrastive Transformer for Tabular Remote Sensing Data

Tanpia Tasnim, Md. Ekramul Islam, [0.5em] Lane Dept. of Computer Science, Electrical Engineering, Dept. of Geology & GeographyWest Virginia University, Morgantown, WV 26506, USA, [0.5em] Department of Computer Science, EngineeringGreen University of Bangladesh, Narayanganj-1461, Bangladesh [email protected], [0.5em] Department of Computer Science, EngineeringStamford University Bangladesh, Dhaka-1217, Bangladesh [email protected] · 2026

Read ZAYAN: Disentangled Contrastive Transformer for Tabular Remote Sensing Data by Tanpia Tasnim, Md. Ekramul Islam, [0.5em] Lane Dept. of Computer Science, Electrical Engineering, Dept. of Geology & GeographyWest Virginia University, Morgantown, WV 26506, USA, [0.5em] Department of Computer Science, EngineeringGreen University of Bangladesh, Narayanganj-1461, Bangladesh [email protected], [0.5em] Department of Computer Science, EngineeringStamford University Bangladesh, Dhaka-1217, Bangladesh [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27606.txt.

Introduction

Bin Habib and three colleagues present ZAYAN, a self-supervised contrastive framework for tabular remote sensing data. ZAYAN learns at the feature level rather than the sample level, needing no explicit anchors and no class labels, while building a disentangled, redundancy-minimized embedding space. It pairs a pretraining module, ZAYAN-CL, with a Transformer, ZAYAN-T, for downstream classification. Across eight data sets, ZAYAN beats tabular deep learning baselines, holding up under label scarcity and distribution shift.

Catalog record

CREATORTanpia Tasnim, Md. Ekramul Islam, [0.5em] Lane Dept. of Computer Science, Electrical Engineering, Dept. of Geology & GeographyWest Virginia University, Morgantown, WV 26506, USA, [0.5em] Department of Computer Science, EngineeringGreen University of Bangladesh, Narayanganj-1461, Bangladesh [email protected], [0.5em] Department of Computer Science, EngineeringStamford University Bangladesh, Dhaka-1217, Bangladesh [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE88874
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27606

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ZAYAN: Disentangled Contrastive Transformer for Tabular Remote Sensing Data Thanks: This paper has been accepted for presentation at the 28th International Conference on Pattern Recognition (ICPR 2026) in Lyon, France. Code: https://github.com/zadid6pretam/ZAYAN . PyPI: pip install zayan . Tanpia Tasnim, Md. Ekramul Islam, [0.5em] Lane Dept. of Computer Science, Electrical Engineering, Dept. of Geology & GeographyWest Virginia University, Morgantown, WV 26506, USA, [0.5em] Department of Computer Science, EngineeringGreen University of Bangladesh, Narayanganj-1461, Bangladesh [email protected], [0.5em] Department of Computer Science, EngineeringStamford University Bangladesh, Dhaka-1217, Bangladesh [email protected] Abstract Learning informative representations from tabular data in remote sensing and environmental science is challenging due to heterogeneity, scarce labels, and redundancy among features. We present ZAYAN ( Z ero- A nchor d Y namic fe A ture e N coding), a self-supervised, feature-centric contrastive framework for tabular data. ZAYAN performs contrastive learning at the feature rather than sample level, removing the need for explicit anchor selection and any reliance on class labels, while encouraging a redundancy-minimized, disentangled embedding space. The framework has two modules: ZAYAN-CL, which pretrains feature embeddings via a zero-anchor contrastive objective with dynamic perturbations/masking, and ZAYAN-T, a Transformer that conditions on these embeddings for downstream classification. Across eight datasets (six remote-sensing tabular benchmarks and two remote-sensing-driven flood-prediction tables from satellite and GIS products), ZAYAN achieves superior accuracy, robustness, and generalization over tabular deep learning baselines, with consistent gains under label scarcity and distribution shift. These results indicate that feature-level contrastive learning and dynamic feature encoding provide an effective recipe for learning from tabular sensing data. K eywords Self-Supervised Learning, Contrastive Learning, Feature-Level Contrast, Zero-Anchor Objectives, Redundancy Minimization, Tabular Remote Sensing Data. 1 Introduction Remote sensing and environmental analyses increasingly rely on large-scale tabular datasets derived from satellite imagery, sensor networks, and environmental surveys [ 64 , 29 ] . Such data bring high dimensionality, feature redundancy, measurement noise, and limited labels, which can undermine conventional supervised learning [ 63 , 68 ] . As a result, Self-Supervised Learning (SSL), especially Contrastive Learning (CL), has become attractive for building label-efficient representations [ 10 , 8 ] . Most CL frameworks, however, operate at the instance level with anchor-based objectives and view augmentations, paying less attention to feature-level structure that is central to tabular representation quality [ 55 ] . Remote-sensing SSL methods such as SeCo [ 42 ] , CACo [ 41 ] , MATTER [ 3 ] , SACo+ [ 54 ] , SwiMDiff [ 60 ] , and Cross-Scale MAE [ 58 ] leverage temporal, multimodal, or multi-scale signals to pretrain image encoders, yet they remain instance/patch-centric and do not dynamically encode feature interactions for tabular inputs. Building on recent benchmark studies that systematically compare tabular deep learning and classical models for urban land cover classification [ 57 ] , we extend this line of work to a broader suite of remote-sensing and related datasets and ask: can we design a feature-level SSL framework that explicitly tackles redundancy and heterogeneity in environmental tabular data? We propose ZAYAN ( Z ero- A nchor d Y namic fe A ture e N coding), a self-supervised Transformer framework that integrates feature-level CL. ZAYAN employs a zero-anchor objective that contrasts features rather than samples, removing the need for explicit anchors and labels. The framework has two components: (1) ZAYAN-CL, a pretraining module that learns informative, disentangled, and redundancy-reduced feature embeddings using a feature-level InfoNCE loss with dynamic perturbations/masking and a redundancy penalty; and (2) ZAYAN-T, a downstream Transformer that consumes these embeddings for supervised classification while preserving the angular/structural relations established during pretraining. Intuitively, ZAYAN encourages each feature to be close to its perturbed views while pushing apart redundant or uninformative directions, yielding a representation where environmental signals of interest are amplified and noise is suppressed. Our contributions are: C1 ZAYAN-CL (feature-level contrastive pretraining). A feature-level, zero-anchor contrastive pretraining scheme that yields robust, disentangled, and redundancy-minimized embeddings without anchors or labels for tabular remote sensing data. C2 ZAYAN-T (Transformer backbone). A Transformer backbone that leverages the pretrained feature embeddings and preserves their structural and angular geometry for improved supervised classification. C3 Comprehensive remote-sensing benchmarks. An extensive evaluation on eight tabular sensing datasets spanning remote-sensing benchmarks and relevant flood-event classification, derived from satellite, GIS, environmental, and socio-economic indicators, building on and extending prior tabular benchmark studies in this domain [ 57 ] . C4 Robustness and deployment-oriented gains. Consistent gains over traditional machine-learning and tabular deep-learning baselines in predictive performance, robustness, and generalization, including under label scarcity, distribution shift, and deployment-oriented diagnostics (calibration, OOD behaviour, and triage-style evaluation). 2 Related Work We review SSL for remote sensing, Transformer and other deep models for tabular data, and applied machine learning for image-derived remote sensing and flood prediction. A. SSL in remote sensing. Remote-sensing SSL has progressed from patch/view-level contrast to spatio-temporal and multi-scale pretraining. Early contrastive methods align augmented views or seasonal variants of the same scene patch (e.g., CMC [ 55 ] , SeCo [ 42 ] ) and incorporate semantic or multispectral cues (SACo+ [ 54 ] ). MATTER refines material/texture representations for downstream classification [ 3 ] . Masked reconstruction and cross-scale design further improve pretraining (Cross-Scale MAE [ 58 ] , SatMAE [ 11 ] ), while diffusion-based matching captures global context (SwiMDiff [ 60 ] ). Geography-aware SSL injects spatial coordinates to respect geospatial relations [ 7 ] . To address scarcity and shift, few-shot and change-aware schemes adapt pretrained encoders or mine challenging positives (RS-FewShotSSL [ 4 ] , CACo [ 41 ] ). These advances largely remain instance/patch-centric and do not explicitly model feature-level interactions in tabular inputs. B. Transformer models for tabular data. Transformers have been adapted to structured data via attention over features and tokens. TabNet uses sequential attentive masks to select salient features [ 6 ] . TabTransformer embeds categorical fields as tokens and attends over feature dependencies before fusing with numerical variables [ 28 ] . FT-Transformer treats all features as tokens with regularized attention and virtual tokens [ 18 ] . SAINT augments Transformer blocks with inter-sample contrastive/triplet objectives [ 52 ] , and AutoInt targets higher-order feature interactions via multi-head attention [ 53 ] . Data-prior pretraining with TabPFN and TabPFN v2 trains compact Transformers on massive synthetic tables for few-shot inference [ 26 , 27 ] . Despite strong results, these methods typically operate at the sample level, treat features uniformly as tokens, and do not directly disentangle redundant feature dimensions. C. Other tabular deep learning and classic models. Beyond Transformers, a growing family of specialized tabular architectures remains highly competitive. Recent nearest-neighbour–aware models such as TabR [ 20 ] and the parameter-efficient ensemble TabM [ 21 ] push state-of-the-art performance by combining strong tabular backbones with kNN-style refinement and lightweight ensembling. TabICL [ 48 ] proposes a tabular foundation model for in-context learning on large collections of tables, while TANDEM [ 44 ] leverages hybrid autoencoders and model-based augmentation to improve robustness in low-label regimes. ProtoGate [ 31 ] introduces prototype-based networks with global-to-local feature selection tailored to biomedical data, and NLP-inspired sequence models such as TabulaRNN [ 59 ] further explore recurrent and RNN-style designs for tabular deep learning. In parallel, earlier architectures like TabSeq, which imposes a sequential feature

Read the complete text