Low-Data Supervised Adaptation Outperforms Prompting for Cloud Segmentation Under Domain Shift

Harshith Kethavath, Weiming Hu · 2026

Read Low-Data Supervised Adaptation Outperforms Prompting for Cloud Segmentation Under Domain Shift by Harshith Kethavath, Weiming Hu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-08956.txt.

Introduction

Kethavath and Hu test whether prompting or light fine-tuning adapts vision-language models better to satellite cloud mapping, where both pictures and wording sit far outside natural-image training. Prompting, the usual route, assumes clever wording can steer a frozen model, but their experiments show small amounts of labeled data win instead. Low-rank fine-tuning beats prompting by 0.03 to 0.09 on the overlap score. The widest gaps fall on spectrally ambiguous cloud types, though tiny labeled sets first dip in performance before recovering.

Catalog record

CREATORHarshith Kethavath, Weiming Hu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34660
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.08956

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Low-Data Supervised Adaptation Outperforms Prompting for Cloud Segmentation Under Domain Shift Harshith Kethavath, Weiming Hu Abstract Adapting vision-language models to remote sensing imagery presents a fundamental challenge: both the visual and linguistic distributions of satellite data lie far outside natural image pretraining corpora. Despite this, prompting remains the dominant deployment paradigm, driven by the assumption that domain-specific language can guide frozen model representations toward specialized tasks. We test this assumption directly on a domain where the mismatch is prominent: cloud segmentation for satellite imagery. Using CLIPSeg on the CloudSEN12+ cloud segmentation benchmark, we evaluate 60 prompt variants spanning simple labels, domain terminology, appearance descriptors, and contextual cues, finding that every variant underperforms the zero-shot baseline (0.255 mIoU), with engineered prompts scoring as low as 0.07 mIoU. No amount of linguistic refinement bridges the gap between CLIP’s natural image representations and satellite spectral imagery. In contrast, supervised fine-tuning with just 0.1% labeled data ( ∼ \sim 8 images) surpasses zero-shot performance overall, and 5–10% data recovers ∼ \sim 85% of maximum achievable mIoU. Full fine-tuning consistently outperforms low-rank adaptation by 0.03–0.09 mIoU, with the largest gaps for spectrally ambiguous classes, and at 0.5 to 1% labeled data, fine-tuning temporarily degrades performance on these classes before recovering, a supervision dip that aggregate mIoU can mask. For practitioners adapting vision-language models to specialized imagery, our results deliver a clear message: labeled data is not the expensive alternative to prompting; it is the worthwhile path. Our code is available at https://github.com/uga-gaim/2026_CVPRW_C loudPrompts 1 Introduction Cloud detection from satellite imagery is a common task in Earth observation, yet deploying modern vision-language models for this task presents a critical assumption buried in the dominant AI deployment paradigm. A recent large-scale study found that 70% of production AI systems rely on prompting models rather than weight tuning 15 . This preference assumes that pretrained representations are close enough to the target domain that language can bridge the remaining gap. For natural images, this assumption often holds. For satellite imagery, we show it does not. Satellite observations differ from natural photographs in fundamental ways. Visually, overhead perspectives, multi-spectral sensors, and amorphous atmospheric phenomena, like clouds that blend into haze, shadows without hard edges, bear little resemblance to the object-centric natural scenes that dominate vision-language pretraining. Linguistically, the gap is equally severe; meteorological vocabulary like “optically thin cirrus” or “cloud shadow” rarely appears in the image caption pairs used to train CLIP-based models. This dual shift creates a compound mismatch that prompting alone cannot resolve. We investigate this failure mode using CLIPSeg 13 , a promptable segmentation model trained on PhraseCut 20 , a dataset of natural images annotated with phrases like “the brown dog” or “leftmost chair”, and evaluate pretrained and fine-tuned variants of the model on CloudSEN12+ 1 , the largest expert-labeled cloud segmentation dataset for Sentinel-2 satellite imagery. Our central question: when severe domain shift is present, can prompt engineering alone compensate for it, or is supervised fine-tuning necessary considering various levels of annotation cost? We conduct a controlled empirical comparison across 60 prompt variants, Low-Rank Adaptation, and Full Fine-Tuning across data budgets from 0.1% to 100%. The answer is unambiguous: every engineered prompt underperforms simple label baselines, while supervised fine-tuning with just 8 labeled images surpasses zero-shot performance on average. Small labeled datasets are not a last resort; they are the right first choice when severe domain shift is present. Our contributions are therefore threefold: 1. We establish that linguistic refinement cannot compensate for fundamental visual-linguistic domain shift, providing the first systematic evidence of this failure for satellite segmentation. 2. We identify a surprisingly low supervision crossover point, as few as 0.1% labeled data ( \sim 8 images) suffice to outperform any prompt strategy, making the case for zero-shot deployment difficult to justify. We further identify a supervision dip phenomenon, at 0.5–1% labeled data, fine-tuning temporarily degrades performance on spectrally ambiguous classes (thin cloud, cloud shadow) before recovering at 2.5–5%, revealing that aggregate mIoU can mask class-level harm when annotation budgets are extremely tight. 3. We show that the choice between Low-Rank Adaptation and Full Fine-Tuning is not a compute tradeoff but a task structure decision: spectral ambiguity, not data volume, determines where each method succeeds. 2 Related Work 2.1 Prompt Engineering Prompt engineering for vision-language models has been studied extensively in image classification, where template-based prompting consistently outperforms raw label prompts 16 . Learnable prompt methods such as CoOp and CoCoOp extend this by optimizing prompt tokens end-to-end, achieving strong generalization across classification benchmarks 23 ; 6 . DenseCLIP has begun extending prompt conditioning to dense prediction 17 , though systematic evaluation for segmentation remains limited. The existing literature leaves two areas under-investigated. First, prompt learning evaluations nearly always operate within natural image domains, testing generalization to novel classes or related datasets but not to fundamentally different visual modalities. Second, no prior work systematically evaluates prompt engineering for segmentation under the dual visual-linguistic shift of satellite imagery, where overhead perspectives, spectral sensors, and meteorological vocabulary all diverge from pretraining distributions simultaneously. We fill both gaps, and find that failure under this shift is total and consistent. 2.2 Supervised Adaptation Supervised adaptation of models spans a spectrum from Full Fine-Tuning, which updates all parameters for maximum flexibility at the cost of compute and forgetting risk 7 , to parameter-efficient methods like Low-Rank Adaptation, which inject low-rank updates into transformer layers while keeping backbone weights frozen 4 . The performance tradeoffs between these approaches vary by domain and data regime 19 . Empirically, Low-Rank Adaptation has demonstrated competitive performance with Full Fine-Tuning on dense prediction tasks including segmentation, with recent work showing it can match unconstrained optimization when adaptation targets are within the low-rank subspace’s expressive range 7 . Low-Rank Adaptation’s adoption has been further driven by minimal inference overhead; updates merge directly into pretrained weights, and a compact hyperparameter space that facilitates systematic search 2 . What remains unclear is how these adaptation strategies compare across varying annotation budgets (availability of labelled data) for vision-language segmentation in remote sensing. This is a domain where visual and linguistic distributions diverge substantially from pretraining data. Prior work does not establish where the performance crossover from zero-shot to supervised adaptation occurs, nor how much labeled data each method requires before gains saturate. Our experiments directly address both questions. 2.3 Domain Shift in Remote Sensing Vision-language models learn joint representations by aligning images and text during pretraining, but this alignment is distribution specific. CLIP’s contrastive objective trains on natural image-caption pairs which reflect the associations between visual patterns and linguistic concepts. When target domains diverge substantially from pretraining distributions, the learned alignment may not transfer, even if underlying visual representations remain useful. This has motivated domain-specific variants such as RemoteCLIP, RS-CLIP, and SenCLIP 9 ; 12 ; 5 . However, such adaptation often requires substantial data curation and computing resources that might be unavailable to practitioners developing the models. A complementary question remains unanswered: for practitioners without access to domain-adapted pretraining, what is the most effective adaptation strategy given a fixed annotation budget? Prior work does not characterize whether prompt engineering can bridge the gap for existing models, nor at what data threshold supervised adaptation becomes worthwhile. Our work

Read the complete text