SAM-CLIP-Thermal: Leveraging large multimodal models for reliable and scalable annotation in thermal image segmentation for field plant phenotyping Authors: Yiyuan Lin, Chenjiao Tan, Changying Li, Yu Jiang Published: 2026 ABSTRACT AbstractThermal imaging enables non-invasive assessment of canopy temperature, an essential indicator of plant stress, yet the lack of color cues and strong shadow interference make plant segmentation in thermal images difficult. Recent advances in foundation models have demonstrated improved performance and generalizability across applications, showing promise for domain-specific applications with limited annotated datasets such as plant segmentation in thermal images. This study investigates large multimodal models (LMMs) for thermal image segmentation in plant phenotyping. Building upon the SAM-CLIP framework, we design a unified pipeline spanning zero-shot inference, few-shot and low-shot fine-tuning, and active learning to maximize accuracy with minimal supervision. Evaluations on two thermal datasets, LadyBird Brassica and UGA Brassica, demonstrate robust performance after minimal adaptation across both datasets and superior performance compared with baselines, achieving mIoUD values of 97.54% on the LadyBird dataset and 76.94% on the UGA dataset. We also release the resulting thermal segmentation annotations to support community benchmarking and reproducible research, highlighting the potential of LMMs to enable scalable, high-quality dataset construction for field phenotyping. The released datasets can be found at: https://cornell.box.com/s/dh69xf84464yrc1vlws92l1tflx7qa89.Keywords: Thermal image segmentation, Large multimodal models, SAM, CLIP, Active learning, Transfer learning, Dataset Introduction Plant segmentation in thermal images is essential for measuring canopy temperature and, in turn, interpreting plant physiological status. However, strong shadows, self-occlusion, and non-plant artifacts often obscure the contours of plant objects in thermal imagery, creating substantial challenges for accurate segmentation. These difficulties are further intensified by the lack of chromatic information in thermal images. Over the past decade, deep learning has transformed semantic segmentation in computer vision. Convolutional neural networks (CNNs) and, more recently, transformer-based architectures have achieved state-of-the-art (SOTA) performance across diverse applications such as natural image benchmarks, aerial imagery, medical imaging, and agricultural RGB phenotyping [[1], [2], [3], [4], [5]]. When large annotated datasets and stable visual cues are available, these models effectively capture hierarchical visual features and long-range contextual relationships, enabling highly accurate pixel-level predictions. Their success has established deep learning as the dominant paradigm for image segmentation. Despite these advances, the application of deep learning to plant segmentation in thermal imagery faces two interconnected bottlenecks: annotation scarcity and limited model generalizability. First, large annotated thermal image datasets for plants remain extremely limited, largely because generating reliable pixel-level annotations is inherently difficult and costly. Thermal images often exhibit low contrast, weak structural edges, and temperature gradients that vary continuously across plant surfaces. These characteristics make object boundaries ambiguous, leading to substantial annotation inconsistency even among trained annotators. Prior work has shown that disagreement on thermal canopy boundaries is common under shadowed or partially occluded conditions [6,7], introducing label noise and undermining the reliability of model training and evaluation. Second, segmentation models for thermal images exhibit strong dataset-specific sensitivity. Radiometric appearance varies with sensor calibration, environmental conditions (e.g., ambient temperature, humidity, and wind exposure), and plant characteristics (e.g., leaf age, angle, and canopy density). As a result, models trained on one field, cultivar, or sensing system often generalize poorly to others. Unlike RGB imagery, where chromatic features remain largely stable across datasets (e.g., different plant species or the same plant species under different environments), thermal image intensities reflect absolute radiance, making them sensitive to subtle shifts in acquisition conditions. These discrepancies significantly limit the transferability of task-specific CNN and transformer-based models and hinder reproducibility across field sites. These challenges not only limit segmentation performance but also impede the creation of large-scale annotated thermal image datasets, which are essential for developing robust and transferable models. Consequently, efficient approaches for generating reliable annotations have become a critical prerequisite for advancing thermal image-based plant phenotyping. Prior efforts to mitigate these challenges have largely focused on domain adaptation [8], synthetic data generation [9,10], or task-specific fine-tuning [3,11]. While such strategies improve within-domain accuracy, they rarely yield transferable representations for field-collected thermal data. Recently, foundation models have dramatically changed the analysis paradigm for many computer vision tasks [[12], [13], [14], [15]], including large multimodal models (LMMs) such as the SAM (Segment Anything Model) [16] and CLIP (Contrastive Language-Image Pre-Training) [17]. Through large-scale pretraining and multimodal alignment between visual and other modalities (e.g., text), these models exhibit strong cross-domain generalization and prompt-based segmentation capabilities under minimal supervision. These properties raise a compelling possibility for agricultural imaging: instead of relying solely on manual annotation, large multimodal models may assist in generating segmentation masks that can serve as annotations for large-scale dataset construction. However, foundation models are predominantly trained on visible-spectrum imagery, while thermal images lack the chromatic and semantic cues that these models typically rely on. Consequently, simply applying these models to thermal imagery collected in agricultural applications often yields unsatisfactory segmentation results. This gap raises two important questions. First, can large multimodal models be effectively adapted to produce reliable segmentation masks for thermal plant imagery with limited supervision? Second, if model-generated masks are used as annotations, how reliable and consistent are these annotations compared with those produced by human annotators? Answering these two questions is essential for establishing scalable annotation workflows and enabling the construction of large benchmark datasets for thermal image-based plant phenotyping. The overarching goal of this study was to develop an efficient annotation framework that integrates foundation models with an active-learning strategy to generate accurate and reliable annotations for thermal images in plant phenotyping. The proposed framework leverages the representation capacity of pretrained foundation models, combined with limited target-domain supervision and active learning, to assist annotation generation while minimizing human labeling effort. Specific objectives were to i) investigate the adaptability of large multimodal models to generate reliable segmentation masks for plant thermal images under limited supervision; ii) quantitatively evaluate the reliability and consistency of model-derived masks compared with human annotations; and iii) demonstrate the efficacy of the developed pipeline to construct and release two large annotated thermal image datasets of plants for future model development and benchmark evaluation. Materials and methods We developed a unified pipeline that combines a custom SAM-CLIP model [18,19] and human-in-the-loop (HITL) active learning to improve both model performance and annotation quality progressively (Fig. 1(B)). In addition to evaluating segmentation accuracy, we further examine the reliability of model-generated masks by comparing them with human annotations and analyzing annotation consistency across multiple annotators. This analysis provides insight into whether model-generated annotations can achieve comparable or even higher reproducibility than manual labeling under challenging thermal imaging conditions. Overview of the thermal datasets and the model development workflow. (A, D): Representative thermal images and corresponding segmentation masks across growth stages in the LadyBird and UGA Brassica datasets. (B): HITL active learning pipeline for iterative annotation and model refinement. (C): Cross-dataset transfer from LadyBird to UGA under zero-shot, few-shot, low-shot, and full fine-tuning regimes. (E): SAM-CLIP architecture for thermal image segmentation. To ensure broad applicability, we evaluated our pipeline on two thermal datasets that represent complementary field conditions: the LadyBird Brassica dataset [20] collected under controlled illumination with clearer canopy boundaries, and the UGA Brassica dataset [21] captured under highly variable lighting and heavy shadowing. This contrast provides a rigorous testbed for assessing model robustness in both favorable and challenging thermal environments. Datasets To evaluate both annotation reliability and model adaptability under diverse thermal imaging conditions, we used two complementary thermal plant datasets. The LadyBird Brassica dataset [20] contains synchronized RGB and thermal imagery acquired under relatively controlled illumination, providing clearer plant boundaries and lower annotation ambiguity. In contrast, the UGA Brassica dataset [21] consists of thermal imagery only and was collected under complex field conditions with strong shadows and substantial visual interference. Together, these two datasets provide a comprehensive testbed for assessing the proposed pipeline in both favorable and challenging scenarios, including cross-dataset transfer under limited supervision and annotation consistency under varying levels of visual ambiguity. Importantly, the two datasets also represent complementary annotation settings: LadyBird corresponds to an RGB-assisted thermal annotation scenario, whereas UGA represents a more challenging thermal-only annotation scenario in which auxiliary RGB references were unavailable. LadyBird dataset The publicly available LadyBird dataset [20] captures the ten-week growth cycle (October–December 2017) of cauliflower and broccoli crops cultivated at the University of Sydney's Lansdowne Farm in Cobbitty, NSW, Australia. The dataset comprises weekly scans collected by an autonomous agricultural robot and includes stereo RGB, thermal, and hyperspectral imagery. In this study, we focused exclusively on the thermal modality, which was acquired using a Xenics Gobi-640 thermal camera equipped with a 10 mm lens. The camera provided a cross-track field-of-view (FOV) of 56.84°, covering approximately 807.26 mm of the crop bed width at the expected imaging height. To improve annotation reliability, thermal images were selected only when their acquisition timestamps were within 0.05 s of the corresponding RGB image timestamps (Supplementary Table S1). The synchronized RGB images were used as references during annotation, which helped reduce ambiguity caused by background elements, shadows, and weeds in low-resolution thermal imagery. To ensure compatibility with standard deep learning models, which typically require 8-bit three-channel inputs (i.e., uint8), the raw 16-bit single-channel thermal images (uint16) were preprocessed by scaling pixel values to 8-bit and duplicating the single channel to form three-channel grayscale images. This standardized preprocessing was applied to all thermal images used in this study. An initial subset of 200 synchronized thermal images, denoted as LB-200 (Supplementary Table S2), was randomly selected with uniform coverage across the first, fifth, and tenth weeks of the collection period. These images were manually annotated for binary plant segmentation, with the foreground class corresponding to brassica and the background class corresponding to all remaining pixels. The LB-200 subset was split into training, validation, and testing sets at a ratio of 6:2:2. A second dataset, LB-450 (Supplementary Table S2), was subsequently constructed through the active learning process described in Section 2.5. UGA dataset The UGA dataset contains six scans acquired during October and November 2017 (Supplementary Table S3). The field was planted with brassica crops at the Horticulture Farm of the University of Georgia in Watkinsville, GA. Six planting rows were scanned during each acquisition using the GPhenoVision system [22] equipped with a thermal imaging module. Thermal images were captured using an A655sc camera (FLIR Systems Inc., Wilsonville, OR, USA), which was mounted at the center of the system at an approximate height of 1 m above the plant bed. The same preprocessing pipeline used for the LadyBird dataset, including bit-depth conversion and three-channel standardization, was applied to the UGA thermal images to ensure input compatibility. To evaluate transfer under limited supervision, we constructed three UGA subsets corresponding to zero-shot, few-shot, and low-shot settings. First, 60 thermal images were randomly selected from the October 12 collection. After excluding images without brassica, 56 images remained. These 56 images were manually annotated for binary plant segmentation and served as the basis for zero-shot, few-shot, and low-shot evaluation (Supplementary Table S4). For zero-shot evaluation, all 56 images were used only for testing. For the low-shot setting, these 56 images were split into training, validation, and testing sets at a ratio of 6:2:2, resulting in the UGA-low-shot subset (Supplementary Table S4). For the few-shot setting, the same validation and testing sets as the UGA-low-shot subset were retained, while a reduced subset of 11 images was randomly selected from the UGA-low-shot training set for model fine-tuning, forming the UGA-few-shot subset (Supplementary Table S4). This design enables a fair comparison between few-shot and low-shot learning by varying only the amount of labeled training data. To improve coverage across growth stages, an additional 30 thermal images were randomly sampled from each of the other five acquisition dates, excluding October 12. After removing images without brassica, 148 annotated images were retained. These images were also split into training, validation, and testing sets using the same 6:2:2 ratio. This multi-date subset was then combined with the previously constructed few-shot subset to form UGA-200. Details of all UGA subsets are summarized in Supplementary Table S4. Global evaluation test sets Because different development rounds involved different training and validation splits, performance measured on round-specific test sets was not directly comparable across the entire model development process. To enable a unified and unbiased evaluation of model quality across stages, we established dedicated global evaluation test sets for both LadyBird and UGA using the same sampling strategy. These global test sets were designed to better reflect overall segmentation quality across the full scope of each dataset rather than within a single split. Specifically, for each dataset, three images were randomly selected from every unique acquisition date and planting row combination. To ensure an unbiased evaluation and eliminate potential data leakage, all selected images were verified to be mutually exclusive from the training, validation, and round-specific test sets used during model development. The selected images were then grouped into three subsets corresponding to early, middle, and late growth stages. As a result, three global subsets were created for LadyBird and three for UGA. These six subsets were used for consistent cross-stage model comparison and for subsequent annotation reliability analysis. Details of the global evaluation test sets are provided in Supplementary Table S5. SAM-CLIP model To address the limited semantics cues of thermal imagery, we adopted SAM-CLIP [18,19] as the core large multimodal model in our pipeline. In this study, SAM-CLIP served not only as the segmentation model for performance evaluation, but also as the annotation engine for generating and progressively refining segmentation masks under limited supervision. As shown in Fig. 1(E), the model integrates two complementary foundation models: SAM, which provides high-resolution structural representations, and CLIP, which contributes semantically grounded multimodal alignment. Given an input image, the SAM image encoder and the CLIP image encoder extract two distinct but complementary visual embeddings. The SAM branch emphasizes fine-grained spatial structure and boundary information, while the CLIP branch captures broader semantic context. In parallel, the CLIP text encoder processes a user-defined text prompt and maps it into the same semantic space as the CLIP visual features. The SAM-CLIP fusion module then projects and combines the SAM and CLIP image embeddings into a shared representation conditioned by the text prompt. The fused representation is finally decoded by the SAM mask decoder to produce pixel-level segmentation outputs. This design is particularly suitable for thermal plant segmentation, where object boundaries are often weak and chromatic cues are unavailable. By combining SAM's strong structural prior with CLIP's semantic guidance, SAM-CLIP provides a practical foundation for adapting visible-spectrum foundation models to thermal imagery with minimal supervision. Throughout training and inference, we used the target plant name (“Brassica”) as the default text prompt. This simple prompt setting enabled language-guided target specification without requiring additional region prompts or manual prompt engineering. Multiple training strategies and fine-tuning configurations, as investigated in our previous work [19], were considered, and the best-performing configuration was adopted as the default SAM-CLIP setting for all downstream experiments in this study. To further evaluate prompt sensitivity, additional experiments using semantically related prompts (“Plant” and “Crop”) under both within-prompt and cross-prompt settings were conducted, with detailed results provided in Supplementary Section 2 (Table S6–S14). Baseline models To comprehensively evaluate the proposed SAM-CLIP framework, we compared it with both conventional semantic segmentation models and recent foundation-model-based segmentation methods. The conventional baselines include UNet [23], KNet [24], and SegFormer [25], representing convolutional, hybrid kernel-based, and transformer-based segmentation architectures, respectively. In addition, three representative foundation-model-based methods were included: SAM [16], IRSAM [26], SAM3 [27], and SAM3-text [27]. SAM serves as the original promotable segmentation foundation model, while IRSAM is an infrared-adapted variant specifically designed for thermal imagery. To evaluate the latest generation of segmentation foundation models, we further include SAM3 in two configurations. SAM3 is fine-tuned as a segmentation model to directly predict masks from input images, whereas SAM3-text extends this baseline by incorporating text prompts during both fine-tuning and inference, enabling multimodal image-text segmentation. This comparison provides a direct assessment of the contribution of semantic text guidance and enables a fair comparison with the proposed SAM-CLIP framework. Together, these methods provide strong baselines for evaluating segmentation accuracy, robustness, and transferability under limited supervision. UNet, KNet, and SegFormer were implemented using the MMSegmentation framework [28]. UNet was configured with an FCN head and pretrained on Cityscapes [29]. KNet employed a Swin Transformer-L backbone with an UperNet decoding head and was pretrained on ADE20K [30]. SegFormer used an MIT-B0 backbone, also pretrained on ADE20K. For the foundation-model-based methods, publicly available pretrained checkpoints were adopted, and all models were fine-tuned under the same training protocol and experimental settings whenever applicable to ensure a fair comparison. Transfer learning To evaluate cross-dataset adaptability under limited supervision, we investigated the transfer of models trained on the LadyBird dataset (LB-200) to the UGA dataset under three settings: zero-shot, few-shot, and low-shot. These settings simulate practical scenarios in which labeled data in the target domain are scarce or unavailable. In the zero-shot setting, the model trained on LB-200 was directly applied to the UGA dataset without any additional fine-tuning. This setting evaluates the intrinsic cross-domain generalization capability of each model. To enable a fair comparison between different supervision levels, the few-shot and low-shot experiments shared the same validation and test sets, differing only in the number of training images used for fine-tuning. In the few-shot setting, the model was fine-tuned using a reduced subset of the UGA-low-shot training set (11 training images in this study) for 10 epochs, while retaining the same validation and test sets as the low-shot setting. This configuration isolates the effect of training data quantity on adaptation performance. In the low-shot setting, the model was fine-tuned using the complete UGA-low-shot training set (34 training images) for 50 epochs, providing a moderate level of supervision for domain adaptation. All fine-tuning experiments were conducted using a consistent learning rate of 1 ×10−4 and a batch size of 8. This experimental design enables a systematic comparison of model transferability from LadyBird to UGA across different supervision levels while ensuring that performance differences are attributable primarily to the amount of labeled training data. Active learning To enable efficient dataset expansion with limited annotation effort, we adopted a HITL active learning strategy to iteratively improve both model performance and annotation quality. Rather than exhaustively annotating all remaining unlabeled thermal images, model-generated segmentation masks were first used as preliminary annotations, allowing human effort to be concentrated on samples where segmentation errors were most evident. Specifically, the best-performing model from the initial training round (trained on LB-200) was applied to the remaining unlabeled LadyBird thermal images to generate preliminary segmentation masks, as illustrated in Fig. 1(B). The resulting segmentation outputs were then manually reviewed, and images exhibiting visually apparent segmentation errors were selected for refinement. Examples of such deficiencies included incomplete plant coverage, inaccurate object boundaries, fragmented predictions, and obvious false positive or false negative regions. Through this visual inspection process, 248 challenging samples were identified for further annotation. This process used manual HITL refinement rather than automated uncertainty-driven sampling. Therefore, no predefined quantitative threshold, uncertainty score, confidence metric, or ranking metric was used for sample selection. Instead, samples were selected based on whether segmentation errors were visually observable and likely to affect annotation quality. For the selected images, annotators refined the model-generated masks rather than creating annotations from scratch. This model-assisted labeling strategy reduced the amount of manual delineation required while ensuring annotation quality. After manual correction, 248 high-quality annotated images were obtained. These samples were split using the same 6:2:2 ratio and combined with the original LB-200 dataset to form the expanded LB-450 dataset (Supplementary Table S2), which was used for subsequent model fine-tuning. To further characterize the annotation effort, we recorded the average annotation time during different stages of dataset construction. During the initial manual annotation stage, 200 LadyBird images were annotated from scratch, requiring an average of 7 min and 21 s per image. After the first round of active learning, model-generated masks were provided as pre-labels for the selected samples with suboptimal predictions, and annotators primarily corrected erroneous regions rather than delineating complete masks from scratch. Under this model-assisted refinement setting, the additional 248 LadyBird images required an average correction time of 3 min and 17 s per image. This active learning process enabled efficient construction of a larger and higher-quality annotated dataset by combining model-assisted labeling with targeted human correction, reducing the average manual annotation time while improving dataset consistency and coverage. Because the objective of this study was dataset construction and annotation refinement rather than benchmarking active learning algorithms, we did not conduct a controlled comparison with random sampling or automated uncertainty-based sampling strategies. Model-derived annotation of the LadyBird and UGA datasets To construct large-scale annotated datasets without additional manual labeling, the best-performing SAM-CLIP model, as determined by the global evaluation test sets, was used as an annotation engine to generate segmentation masks for the full LadyBird and UGA datasets. For each dataset, the trained model was applied to all available thermal images to produce pixel-level plant segmentation masks. These model-generated masks served as the primary annotations for dataset expansion. To ensure annotation quality and dataset usability, post-processing was performed to remove degenerate cases. Specifically, masks containing only background or only foreground pixels were excluded, as such samples do not provide meaningful supervision for segmentation tasks and may introduce bias during training. After filtering, the retained masks constituted the final model-derived annotation sets for both datasets. To further characterize the properties of the generated annotations, foreground area ratios were computed for all images. These statistics provide a quantitative description of object coverage across growth stages and serve as a basis for subsequent analysis of dataset distribution and annotation consistency. This model-driven annotation process enables scalable and consistent dataset construction, particularly for thermal imagery where manual labeling is difficult and subject to ambiguity. By leveraging foundation models as annotation generators, the proposed pipeline significantly reduces annotation cost while maintaining high-quality segmentation labels. Human annotation collection and consistency analysis using global evaluation test sets To evaluate annotation reliability and quantify human variability under different levels of visual ambiguity, we designed two complementary annotation protocols based on the LadyBird and UGA global evaluation test sets (Section 2.1.3). These protocols were used to characterize inter-annotator variability, and the resulting ground truth (GT) labels, constructed under the defined annotation procedures, were subsequently used to assess the quality and consistency of model-generated masks. For each dataset, each subset was assigned to four human annotators, including one subset lead and three independent annotators. The subset leads were experienced annotators who had participated in earlier stages of dataset development and were familiar with the labeling protocols and quality control procedures. To improve annotation efficiency while preserving annotator independence, SAM-CLIP-generated masks were provided as optional visual references. Annotators were not required to follow these pre-labels and could freely modify or disregard them based on their own judgment. All annotations were performed on identical thermal image sets using the same labeling interface and guidelines to ensure consistency across annotators. LadyBird dataset annotation procedure The annotation protocol for the LadyBird dataset was designed to produce high-quality GT labels under relatively low visual ambiguity. Each thermal image was accompanied by a synchronized RGB image, which served as a reference to improve boundary identification and reduce uncertainty. For each subset, four annotators were assigned, including one subset lead and three independent annotators. After the initial annotation phase, the subset lead reviewed all submitted masks, resolved discrepancies, and performed necessary corrections to ensure internal consistency. The reviewed masks were then consolidated as the final GT labels for each subset. This protocol combines independent annotation with expert-guided refinement, resulting in high-quality and relatively stable GT annotations. UGA dataset annotation and consistency analysis The UGA dataset was used to assess annotation consistency under more complex and visually ambiguous conditions. Unlike the LadyBird dataset, UGA thermal images lacked RGB references and were affected by strong shadows and confounding visual elements such as paper tags that resemble plant structures. A total of 12 annotators participated in this task. Three experienced annotators served as subset leads for the three subsets (UGA G1–G3) and produced the GT annotations used for model evaluation. The remaining nine annotators were randomly assigned to the subsets, with three annotators per subset. Each annotator independently labeled all 36 images within their assigned subset. All annotations were collected without any post hoc correction or consensus enforcement in order to preserve the original human variability. This design enables a direct and unbiased assessment of inter-annotator disagreement and provides a realistic estimate of annotation uncertainty in challenging thermal imagery. To further evaluate the reliability of model-generated masks, SAM-CLIP predictions were incorporated as an additional annotator in subsequent analyses. This allows direct comparison between human-only consistency and human-plus-model consistency, providing insight into whether model-generated annotations can offer more stable and reproducible labeling for large-scale dataset construction. Evaluation metrics A comprehensive model performance evaluation strategy was adopted, integrating dataset-level, image-level, and worst-case metrics to fully characterize performance in the binary segmentation task (Table 1). Since the task focuses solely on the object class (brassica in this study), all reported metrics were computed exclusively for this target class. Although metric values calculated on the target class (foreground) alone are typically lower than overall two-class metrics (which include the background), they more accurately reflect the segmentation performance of interest. Table 1Overview of evaluation metrics.Table 1MetricPurposeEvaluated onmIoUDDataset-level accuracyModelmIoUIImage-level accuracyModelmIoUCqRobustness under difficult samplesModelPairwise mIoUOverlap between annotatorsHumanFleiss' κMulti-annotator agreementHuman and ModelBinary entropyPixel-level uncertaintyHuman Overview of evaluation metrics. In addition to evaluating model performance, we analyzed the reliability and consistency of human annotations and compared them with the model-generated masks used to construct the datasets. Because large-scale manual annotation of thermal imagery is both time-consuming and subjective, quantifying the degree of variance among human annotators is crucial for assessing dataset quality and reproducibility. Substantial disagreement among annotators may indicate the presence of noisy or inaccurate labels, which can negatively affect subsequent model training and evaluation. To quantify this variability, annotation consistency was assessed using three complementary metrics: the pairwise mean intersection over union (pairwise mIoU) for measuring direct overlap between human annotations, Fleiss’ Kappa (κ) for evaluating multi-annotator agreement beyond chance, and binary entropy for quantifying pixel-level disagreement among annotators (Table 1). Collectively, these metrics provide a comprehensive evaluation of annotation reliability and enable a direct comparison between human and model consistency. Dataset-level mean intersection over union (mIoUD) The dataset-level mean Intersection over Union, mIoUD, provides a macro-level view of model accuracy across the entire test set. It is computed by aggregating all predictions and GT labels into a global confusion matrix before calculating the Intersection over Union (IoU) for each class. For a binary segmentation task, the IoU for the target class c is defined as: where TPc, FPc, and FNc denote the total number of true positives, false positives, and false negatives, respectively, accumulated across all images in the dataset. The reported metric for the target class is then: Since mIoUD is weighted by pixel count, larger images contribute more to the final value, which is particularly relevant when object sizes vary significantly across the dataset. Image-level mean intersection over union (mIoUI) The image-level mean Intersection over Union, mIoUI, is obtained by computing the IoU individually for each image and then averaging the results over the dataset. This approach mitigates the bias toward large images and better reflects prediction consistency at the image level. For image i and class c, the IoU is: where TPi,c, FPi,c, and FNi,c are the confusion matrix elements for image i. The target class mIoUI across N images is: In this formulation, each image contributes equally to the overall score, regardless of resolution or target object size. Worst-case metrics To quantify model robustness and identify potential failure cases, worst-case metrics were computed by focusing on the worst-performing samples. This analysis calculates the mean IoU over the subset of images exhibiting the poorest performance. For the target class, IoUi,target is first computed for every image i. These IoU values are then sorted in ascending order, and the worst q% of samples (e.g., q = 1, 5, 10) are selected. The worst-case mean IoU is defined as: where K = ⌊N × q/100⌋ is the number of images in the selected subset and N is the total number of images in the test set. By isolating the most challenging samples, these metrics provide insights into the model's robustness and its reliability under difficult conditions. Pairwise mean intersection over union Pairwise mean intersection over union (pairwise mIoU) was used to evaluate the degree of overlap between annotation masks produced by different human annotators within each subset. The metric follows the same definition as introduced in Section 2.8.1 and Section 2.8.2, but here it was computed exclusively between human annotations rather than between model predictions and GT. For each pair of annotators, the pairwise mIoU quantifies the intersection over union of their binary segmentation masks, providing a direct measure of annotation consistency within a subset. Lower pairwise mIoU values indicate greater disagreement among annotators, reflecting subjectivity or uncertainty during manual labeling. Fleiss’ Kappa (κ) To quantify inter-annotator agreement beyond chance at the image level, Fleiss’ Kappa (κ) was computed for each image individually. Unlike pairwise mIoU, which measures agreement between two annotators at a time, κ captures collective consistency among all annotators for the same image. It is defined as: where Po represents the observed proportion of agreement among annotators and Pe denotes the expected proportion of agreement by random chance. Both Po and Pe were derived from the pixel-wise label matrix, where each pixel was treated as an item labeled by multiple raters. A κ value of 1 indicates perfect agreement, while κ = 0 corresponds to random labeling. The per-image κ values were then used to analyze the distribution of human annotation consistency across all subsets. To evaluate the consistency of the model-derived masks relative to human annotations, an additional set of κ values was computed by treating the SAM-CLIP prediction as an extra annotator and recomputing the group-level agreement. Binary entropy Binary entropy was used to quantify pixel-level disagreement among annotators. For a binary segmentation task, each pixel can be assigned either the foreground or background label by multiple annotators. The entropy H is computed as: where p is the empirical probability that a given pixel is labeled as foreground by the annotators. Entropy reflects the uncertainty of the labeling decision at each pixel. Since each pixel was labeled by four annotators, only three distinct entropy values were possible, corresponding to complete agreement, partial disagreement, and maximum disagreement. Specifically, H equals 0 when all annotators assign the same label (4:0 voting), reaches its maximum when the annotators are evenly divided (2:2 voting), and takes an intermediate value when three annotators agree and one disagrees (3:1 voting). The resulting entropy distribution provides a spatially resolved quantitative measure of local annotation variability, highlighting regions characterized by elevated human uncertainty. Results Comparisons between SAM-CLIP and baseline models The performance on both the LB-450 and UGA-200 datasets demonstrated that SAM-CLIP consistently achieved the best segmentation performance among all compared methods (Fig. 2). On the LB-450 dataset, SAM-CLIP obtained an mIoUI of 85.40% and an mIoUD of 97.54%, outperforming all conventional and foundation-model-based baselines. A similar trend was observed on the more challenging UGA-200 dataset, where all methods showed lower overall performance, yet SAM-CLIP continued to achieve the highest accuracy with an mIoUI of 71.71% and an mIoUD of 76.94%. Notably, SAM-CLIP consistently outperformed the strongest competing foundation model, SAM3, on both datasets, indicating that incorporating language guidance provides additional benefits beyond recent segmentation foundation models. Performance comparison on mIoUs (%) of different models on LB-450 (A) and UGA-200 (B) datasets. The metrics mIoUC1, mIoUC5, and mIoUC10 represent mIoUI computed on the worst-performing 1%, 5%, and 10% of images in each test set. These correspond to 1, 4, and 9 images in LB-450 (A), and 1, 2, and 4 images in UGA-200 (B), respectively. Among these, UNet and SegFormer achieved 0% mIoU on some worst-case metrics. Representative qualitative results are shown in Fig. 3, Fig. 4. Across both datasets, the predictions of SAM-CLIP most closely matched the ground truth while preserving object boundaries and fine structural details. In comparison, UNet frequently missed small plants, KNet occasionally produced incomplete masks or false positives, and SegFormer struggled with small objects and complex boundaries. Among the foundation-model-based baselines, SAM and IRSAM still overlooked small targets under challenging thermal conditions, whereas SAM3 and SAM3-text generated more complete masks but tended to over-segment boundaries or lose fine details. These observations are consistent with the quantitative comparisons. Qualitative comparison of segmentation results from baseline models and SAM-CLIP on the LadyBird dataset. Example images are arranged from top to bottom with increasing plant size, ranging from small isolated plants to dense canopy structures. Qualitative comparison of segmentation results from baseline models and SAM-CLIP on the UGA dataset. Example images are arranged from top to bottom with increasing object size, ranging from small scattered plants to larger canopy regions. SAM-CLIP also consistently achieved the highest performance on the worst-performing subsets of the LB-450 dataset, including the worst-performing 1%, 5%, and 10% of testing samples. Although recent foundation-model-based methods such as SAM3 and SAM3-text demonstrated competitive overall performance, their accuracy degraded more noticeably on the most challenging samples. In contrast, traditional segmentation models exhibited substantially larger performance drops, with UNet and SegFormer even yielding 0% IoU on some worst-case subsets. These results indicate that SAM-CLIP not only improves average segmentation accuracy but also maintains superior robustness under challenging imaging conditions. In addition, supplementary experiments with different text prompts showed that SAM-CLIP remained robust under both within-prompt and cross-prompt settings. Under the best-performing configuration, mIoUI varied only from 90.44% to 90.86% across within-prompt settings, while cross-prompt evaluation still achieved 84.11%–85.47% mIoUI, demonstrating reasonable robustness to prompt selection (Supplementary Section 2). Transfer learning performance Zero-shot transfer yielded 0% for all mIoU metrics across all models. This result highlights the substantial out-of-distribution (OOD) gap between the source and target datasets, which differ in thermal distributions due to variations in imaging sensors, cultivars, and acquisition conditions. The complete failure of zero-shot transfer indicates that thermal image segmentation exhibits limited cross-dataset generalization under severe OOD conditions and requires adaptation to the target domain despite the shared plant structures. To evaluate the amount of target-domain adaptation required under this OOD setting, all models were fine-tuned using the UGA-few-shot and UGA-low-shot datasets. As shown in Fig. 5, SAM-CLIP consistently achieved the highest performance across all evaluation metrics under both transfer learning settings. With only a few annotated training images, SAM-CLIP reached an mIoUI of 67.93% and an mIoUD of 68.85% on the UGA-few-shot dataset while maintaining clear advantages on the most challenging samples, achieving 40.76% and 53.45% mIoUI on the worst-performing 1% and 20% of images, respectively. Increasing the number of annotated training images further improved performance, with SAM-CLIP achieving an mIoUI of 73.17% and an mIoUD of 73.93% on the UGA-low-shot dataset. Few-shot (A) and low-shot (B) performance of different models on the UGA-56 dataset. The metrics mIoUC1, mIoUC20, and mIoUC30 represent mIoUI computed on the worst-performing 1%, 20%, and 30% of images in each test set. These correspond to 1, 2, and 3 images in the test set. These results demonstrate that, although thermal image segmentation shows limited zero-shot generalization across OOD datasets, SAM-CLIP can substantially reduce the annotation effort required for target-domain adaptation. This makes the proposed framework well suited for scalable deployment across diverse thermal phenotyping datasets, where a small amount of target-domain annotation is often more practical than fully manual dataset construction. Large-scale annotation generation and quality evaluation To facilitate large-scale thermal phenotyping research, we generated and publicly released segmentation annotations for the complete LadyBird and UGA thermal image datasets using the best-performing SAM-CLIP model. Specifically, the final model obtained through the multi-round training process was used to generate 20,538 masks for the LadyBird dataset and 37,790 masks for the UGA dataset. Details of the generated annotations are provided in Supplementary Fig. S1, and both annotated datasets are publicly available at: https://cornell.box.com/s/dh69xf84464yrc1vlws92l1tflx7qa89. To assess whether these automatically generated masks are sufficiently reliable for large-scale annotation, we evaluated the model on the global evaluation datasets described in Section 2.1.3. These datasets consist of randomly selected images covering the full data distribution, with manually annotated ground truth provided by experienced annotators. As shown in Table 2, SAM-CLIP achieved consistently high segmentation accuracy across all LadyBird subsets, with mIoUI values of 87.52%, 96.36%, and 97.53%. Even on the most challenging early-growth subset (LB-G1), the model achieved a worst-case mIoUC1 of 74.93% on the worst-performing 1% of images, demonstrating robust performance across different growth stages and object sizes. These results indicate that the generated masks maintain high quality throughout the entire LadyBird dataset. Table 2Mask quality evaluation on the LadyBird global test sets. SAM-CLIP-200 denotes the best-performing model trained on the LB-200 dataset during the initial training round, while the remaining models correspond to those trained using the LB-450 dataset after active learning. mIoUC1, mIoUC5, and mIoUC10 denote mIoUI computed on the worst-performing 1%, 5%, and 10% of images in each subset, corresponding to 1, 2, and 4 images, respectively.Table 2ModelDatasetmIoUImIoUDmIoUC1mIoUC5mIoUC10SAM-CLIP-200LB-G1(obj. ratio:4.23%)83.3487.5441.4243.1752.76SAM-CLIP87.5290.0974.9376.6978.04SAM386.9389.2670.3072.7874.73SAM3-text86.3488.3265.8668.7671.37SAM85.3786.9058.9862.0365.44IRSAM78.4582.6930.6944.2149.78UNet65.0972.9221.4125.3327.76KNet84.7385.9554.8056.9161.14SegFormer42.0058.850.000.000.00SAM-CLIP-200LB-G2(obj. ratio:35.95%)94.7495.2684.8085.0686.66SAM-CLIP96.3696.5990.4691.9993.31SAM396.3196.5690.2791.8593.06SAM3-text96.2796.5290.0991.7192.81SAM96.1996.4789.8091.4892.36IRSAM95.1495.7585.2088.2089.47UNet93.0193.7083.4983.5984.50KNet96.1496.4389.6391.3092.04SegFormer90.3890.9779.7380.8081.58SAM-CLIP-200LB-G3(obj. ratio:50.48%)95.5396.0276.7279.9283.83SAM-CLIP97.5397.5792.0792.4492.85SAM397.3297.4287.1489.7891.71SAM3-text97.1297.2582.4287.0490.56SAM96.7897.0075.1082.4688.53IRSAM94.4495.1555.3169.5177.52UNet89.9490.1249.4155.4559.05KNet96.5696.8370.6578.9787.05SegFormer93.4093.8676.5680.0082.29 Mask quality evaluation on the LadyBird global test sets. SAM-CLIP-200 denotes the best-performing model trained on the LB-200 dataset during the initial training round, while the remaining models correspond to those trained using the LB-450 dataset after active learning. mIoUC1, mIoUC5, and mIoUC10 denote mIoUI computed on the worst-performing 1%, 5%, and 10% of images in each subset, corresponding to 1, 2, and 4 images, respectively. On the UGA global evaluation datasets, SAM-CLIP exhibited similarly strong performance, achieving mIoUI values of 85.41%, 89.74%, and 90.86% across the three subsets (Table 3). More importantly, comparison with independent human annotators (Fig. 6(B)) showed that SAM-CLIP outperformed several annotators across all subsets and even achieved a higher overall mIoUI than every individual annotator. These results demonstrate that the generated masks reach human-level annotation quality and are sufficiently reliable for large-scale dataset construction. Table 3Mask quality evaluation on the UGA global test sets. SAM-CLIP-LS denotes the best-performing model trained on the UGA few-shot dataset, while the remaining models were trained on the UGA-200 dataset. mIoUC1, mIoUC5, and mIoUC10 denote mIoUI computed on the worst-performing 1%, 5%, and 10% of images in each subset, corresponding to 1, 2, and 4 images, respectively.Table 3ModelDatasetmIoUImIoUDmIoUC1mIoUC5mIoUC10SAM-CLIP-LSUGA-G1(obj. ratio:0.74%)77.8278.3458.3763.3565.17SAM-CLIP85.4185.9675.5476.0476.44SAM381.2682.4763.8267.3169.44SAM3-text79.8481.1660.5764.8267.03SAM74.9277.0843.7150.6355.29IRSAM63.4868.2412.4620.8527.62UNet52.5144.830.000.921.41KNet76.1479.2825.0043.4651.04SegFormer53.1747.670.372.224.26SAM-CLIP-LSUGA-G2(obj. ratio:3.47%)59.1449.627.1115.0321.20SAM-CLIP89.7488.7550.5963.1067.96SAM382.6384.9142.1653.8258.74SAM3-text80.1882.3738.4449.2654.81SAM75.4278.5633.1842.7347.96IRSAM66.0773.4224.5131.8636.74UNet55.0268.2822.0626.2628.35KNet72.2681.1045.8646.2349.17SegFormer54.1269.1826.8527.7128.00SAM-CLIP-LSUGA-G3(obj. ratio:4.99%)49.9328.620.000.110.32SAM-CLIP90.8688.8756.1160.9763.86SAM386.4385.6949.2155.1759.94SAM3-text83.7283.6444.8351.0656.72SAM78.5977.6444.8249.3755.21IRSAM69.8472.9331.5738.4244.18UNet51.3252.8620.7023.9725.86KNet71.9473.1751.5952.2854.13SegFormer55.1662.467.7018.1623.83 Mask quality evaluation on the UGA global test sets. SAM-CLIP-LS denotes the best-performing model trained on the UGA few-shot dataset, while the remaining models were trained on the UGA-200 dataset. mIoUC1, mIoUC5, and mIoUC10 denote mIoUI computed on the worst-performing 1%, 5%, and 10% of images in each subset, corresponding to 1, 2, and 4 images, respectively. Analysis of human annotation disagreement on the UGA global test dataset. (A) Distribution of per-image Fleiss' κ across three UGA subsets, comparing human-only annotations with annotations that include the SAM-CLIP prediction. (B) Inter-annotator agreement measured by pairwise mIoUD and mIoUI among human annotators. Human annotation consistency and reliability of model-generated masks To further evaluate the reliability of model-generated annotations, we analyzed annotation consistency on the UGA global test dataset, where each image was independently labeled by four human annotators following the same annotation protocol. Agreement was evaluated using complementary statistical and spatial metrics, including Fleiss’ κ, pairwise mIoU, and binary entropy. In addition, the SAM-CLIP prediction was treated as an additional annotator to assess its influence on overall annotation consistency. Including SAM-CLIP increased group agreement and reduces variance As shown in Fig. 6(A), incorporating the SAM-CLIP prediction as a fifth annotator consistently increased the mean Fleiss’ κ across all UGA subsets while reducing variance (p < 0.001, two-sided t-test). This improvement was observed across all growth stages, indicating that the model-generated masks provide a stable and reproducible reference that strengthens annotation consensus rather than merely serving as coarse pre-labels. Inter-human variability was substantial and depended on growth stage Pairwise mIoU analysis (Fig. 6(B)) revealed considerable variability among human annotators, with agreement ranging from approximately 78% to 95% despite following the same annotation protocol. Agreement was lowest for early-stage plants (UGA-G1), where small targets and shadow interference increase annotation ambiguity, and highest for late-stage plants (UGA-G3), where canopy boundaries are more distinguishable. Compared with the model-GT agreement results in Table 3, SAM-CLIP achieved agreement levels comparable to or exceeding typical inter-human consistency, demonstrating that its predictions are at least as reproducible as manual annotations. Entropy maps localized disagreement to thermally ambiguous regions Binary entropy maps (Fig. 7) further revealed that annotation disagreement is concentrated around object boundaries, shadows, and non-target objects with thermal signatures similar to plant tissue, while interior canopy regions exhibit consistently low entropy. For example, the highest-entropy region in Fig. 7(B) corresponds to a paper plot tag that SAM-CLIP correctly excluded but multiple human annotators mistakenly labeled as plant tissue. Similar observations across other examples suggest that annotation disagreement primarily arises from intrinsic ambiguity in thermal imagery rather than inconsistent labeling behavior. Visualization of annotation disagreement on the UGA global test dataset. Each row shows a thermal image, the SAM-CLIP prediction (used as a pre-label), four human annotations (A1–A4), and the corresponding binary entropy map. In the entropy map, yellow denotes maximum disagreement (2:2 voting), green indicates moderate disagreement (3:1 voting), and purple indicates full agreement (4:0 voting). Implications for scalable dataset construction Together, these analyses show that variability is an inherent characteristic of manual thermal image annotation, even under carefully controlled annotation protocols. By producing predictions with human-level consistency while reducing annotation variability, SAM-CLIP provides a reliable foundation for scalable dataset construction. This capability substantially reduces the dependence on labor-intensive manual annotation and directly supports the development of large-scale thermal phenotyping datasets. Discussion Generalization capability of large multimodal models and a shift in agricultural vision paradigms LMMs offer important advantages over task-specific architectures due to their exposure to diverse visual and linguistic concepts during pretraining, which provides rich transferable representations. However, our zero-shot results also show that such representations do not guarantee unrestricted cross-dataset generalization in thermal plant segmentation. Based on our evaluation on two thermal image datasets, the main advantage of SAM-CLIP lies in its ability to achieve strong performance after limited target-domain adaptation, although it had not been pretrained on thermal imagery. With only a small number of thermal images for fine-tuning, SAM-CLIP achieved performance levels that conventional segmentation models, such as UNet and SegFormer, could not reach under the same data constraints. The superior performance of SAM-CLIP can be attributed to the complementary strengths of its two foundation-model components. SAM contributes strong structural priors and high-resolution boundary representations, enabling the model to capture plant shapes and maintain spatial continuity even when thermal contrast is weak. Meanwhile, CLIP provides semantically meaningful representations learned from pretraining on large-scale image-text pairs, which help distinguish plant structures from visually similar background artifacts after adaptation to the thermal domain. Experimental observations support this interpretation. SAM-CLIP consistently outperformed CNN-based and transformer-based baselines on both datasets, suggesting that its pretrained representations are more transferable under limited supervision. Additionally, the model achieved agreement levels comparable to or exceeding human annotators on the UGA dataset and increased overall inter-annotator consistency when treated as an additional annotator. More broadly, these observations reflect an emerging paradigm shift in agricultural computer vision research. Foundation models, once appropriately fine-tuned, are increasingly capable of supporting a wide range of agricultural applications, even under domain shifts such as non-RGB sensing modalities. In contrast, many highly specialized SOTA architectures, despite their structural sophistication and strong performance on benchmark datasets, often exhibit limited transferability to agricultural scenarios, where data distributions, visual cues, and environmental variability differ substantially from those in mainstream computer vision tasks. Similar observations have been reported in recent agricultural vision studies, where complex architectures show limited transferability under domain shifts [8,31,32], while foundation models demonstrate stronger adaptability after task-specific fine-tuning. This contrast suggests that, at the current stage of agricultural AI development, the marginal benefit of designing increasingly complex network architectures may be limited when compared to the gains achievable through improved data quality, scale, and consistency. Consequently, rather than prioritizing further architectural complexity, this work deliberately selects SAM-CLIP as a representative large multimodal model and focuses on its capacity to generate reliable, large-scale annotations. From this perspective, the primary contribution of the present study is not the proposal of a novel architecture, but the demonstration that LMMs can serve as effective engines for scalable dataset construction in agriculture. By leveraging the generalization capability of foundation models, the bottleneck of limited high-quality annotations can be alleviated, enabling downstream model development and analysis without requiring excessive manual labeling effort. Nevertheless, task-specific segmentation models retain advantages in computational efficiency. They are typically more compact, offer faster inference, and are easier to deploy on resource-constrained edge devices. This highlights a practical trade-off between generalization capability and deployment efficiency, which should be carefully considered when selecting models for field-based agricultural applications. The proposed SAM-CLIP framework is intended for offline large-scale annotation, where annotation quality is prioritized over inference efficiency. By releasing large-scale, high-quality annotated thermal datasets, this work provides a valuable foundation for developing lightweight and deployment-oriented thermal image segmentation models through pretraining or task-specific fine-tuning. Such models are expected to better satisfy the computational requirements of future field applications. Human annotation variability and the role of model-assisted labeling Annotation consistency is highly dependent on image complexity, which poses a major challenge for generating reliable training labels. For demanding segmentation tasks, such as plant disease segmentation in RGB images requiring domain-specific expertise, or thermal imagery collected under uncontrolled field conditions with shadows and interference from surrounding objects, annotation subjectivity becomes unavoidable. This issue is particularly evident in the UGA dataset, where ambiguous thermal boundaries lead to noticeable inter-annotator disagreement. Different human annotators may interpret object boundaries differently, introducing inconsistent masks and ambiguous supervision into the training data. Such annotation noise can adversely affect model generalization, as models may learn conflicting or unstable representations rather than intrinsic object characteristics. To mitigate this issue, ambiguous samples are often subjected to multiple rounds of human review and verification. While this strategy can improve label reliability, it is time-consuming, labor-intensive, and difficult to scale to large datasets. Moreover, even after extensive review, subtle discrepancies among annotators often persist, especially for thermal images where visual cues are inherently weak. Our evaluation comparing model predictions with human annotations indicates that the SAM-CLIP model surpassed several human annotators on the UGA dataset. This result further supports the notion that foundation models can play a central role not only in downstream inference, but also in the data creation process itself. Inter-annotator variability and inconsistent supervision have also been widely observed in prior segmentation studies [33,34], motivating the use of model-assisted or semi-automated labeling strategies to improve annotation consistency and scalability. In challenging agricultural imaging scenarios, model-assisted labeling may offer a level of consistency that is difficult to achieve through manual annotation alone. These findings highlight the potential of LMM-driven annotation workflows to reduce human subjectivity, accelerate labeling, and improve annotation consistency at scale. Value of large-scale model-generated annotations for agricultural research To facilitate further research and model development, we release the thermal images together with the corresponding segmentation masks predicted by the SAM-CLIP model. Given that these generated masks demonstrate quality comparable to human annotations, they can support a range of downstream applications, including automated temperature-based plant stress analysis and disease monitoring under field conditions. Related efforts have shown that model-generated or weakly supervised annotations can serve as effective alternatives to fully manual labeling in agricultural deep learning tasks [[35], [36], [37]] and thermal [38] segmentation scenarios, enabling dataset construction at scale. Importantly, the released dataset reflects a data-centric research philosophy, in which the emphasis is placed on building scalable, reusable, and high-quality annotated resources rather than pursuing incremental architectural complexity. Such datasets can serve as common benchmarks for evaluating future methods and may enable more rapid progress across the agricultural vision community. Failure modes and remaining challenges in thermal segmentation Despite achieving reasonable overall segmentation performance, the SAM-CLIP model exhibits several representative failure cases, as illustrated in Supplementary Fig. S2. On the LadyBird dataset, thin plants with minimal temperature contrast relative to the surrounding soil often result in false negatives across all evaluated models (Fig. S2(A)). Additionally, cooler non-plant objects, such as irrigation tubes beneath the canopy, are occasionally misclassified as plant regions, leading to false positives (Fig. S2(B)). For the UGA dataset, segmentation performance is notably affected by shadows and non-biological objects such as plot tags. Shadows may be incorrectly segmented as plant regions (Fig. S2(C)), while plot tags can also be misclassified due to their distinct thermal signatures. Similar shadow-induced false positives are observed in Fig. S2(D). These failure cases underscore the inherent difficulty of thermal image segmentation, particularly in scenarios where models cannot rely on color or texture cues that are informative in RGB imagery. Similar failure modes caused by low thermal contrast, shadows, and background interference have been reported in prior studies on thermal plant segmentation [39,40], and multimodal RGB-thermal fusion [41] has been proposed as a potential strategy to mitigate these challenges. Future work may benefit from collecting aligned RGB and thermal image pairs to exploit complementary information across modalities and further improve segmentation robustness. Limitations in downstream validation and implications for active learning Due to the unavailability of calibration data and radiometric conversion equations for the thermal cameras used in the LadyBird and UGA datasets, as well as the absence of plant-level identity information, it was not possible to accurately extract canopy temperature for individual plants. Consequently, downstream validation linking segmentation accuracy to canopy temperature estimation could not be conducted in this study. Without radiometric calibration, the practical impact of marginal segmentation improvements cannot be directly quantified. If a certain level of segmentation accuracy is already sufficient to support reliable canopy temperature extraction, further pursuit of marginal gains in mIoU may yield diminishing returns. Under such conditions, evaluating an extensive set of increasingly complex SOTA architectures becomes less informative, as small performance differences cannot be meaningfully translated into downstream phenotyping benefits. This consideration partly explains why the present study focuses on a limited yet representative set of segmentation models including SAM-CLIP as a large multimodal foundation model, together with UNet, KNet, and SegFormer as widely adopted task-specific baselines. Rather than exhaustively benchmarking numerous SOTA variants, our emphasis is placed on understanding whether foundation models can reach a performance regime that is already sufficient for downstream thermal phenotyping, and on assessing their potential to support scalable data generation and annotation. Conclusion This study developed a data annotation pipeline based on a large multimodal model and active learning for plant thermal imagery. The SAM-CLIP model achieved the highest segmentation performance on two thermal datasets, surpassing all semantic segmentation baselines. Additionally, the few-shot and low-shot fine-tuning results on the UGA dataset showed that the proposed framework can substantially reduce the amount of target-domain annotation required to transfer the model to a new thermal dataset. The comparison between model predictions and human annotations further indicated that SAM-CLIP can not only assist annotation but also achieve a level of consistency and precision comparable to that of human annotators. The resulting thermal segmentation annotations of both LadyBird Brassica and UGA Brassica datasets have been released to support model benchmarking and the development of phenotyping tools. Overall, this pipeline offers a promising direction toward efficient and scalable annotation of thermal imagery for plants, potentially accelerating large-scale dataset development and enabling downstream applications such as automated temperature-based plant stress analysis and disease monitoring. CRediT authorship contribution statement Yiyuan Lin: Conceptualization, Data curation, Methodology, Software, Writing - original draft, Writing - review & editing. Chenjiao Tan: Conceptualization, Data curation, Methodology, Software, Writing - original draft, Writing - review & editing. Changying Li: Conceptualization, Data curation, Methodology, Resources, Supervision, Writing - review & editing. Yu Jiang: Conceptualization, Data curation, Methodology, Resources, Supervision, Writing - review & editing. Data availability The dataset can be accessed at: https://cornell.box.com/s/dh69xf84464yrc1vlws92l1tflx7qa89. Funding Authors Y.L. and Y.J. were supported by the Specialty Crop Research Initiative Program of the USDA National Institute of Food and Agriculture (Award No. 2022-51181-38240). Authors C.T. and C.L. were supported by the USDA National Institute of Food and Agriculture (Award No. 2023-67021-40646). Declaration of competing interest Given his role as Associate Editor, Yu Jiang had no involvement in the peer review of this article and had no access to information regarding its peer review. Full responsibility for the editorial process for this article was delegated to another journal editor. Acknowledgments The authors sincerely thank the annotators for their careful and consistent manual labeling of the dataset, which enabled the quantitative evaluation presented in this study, including Adam Camerer, Ruiming Du, Nicolas Gunner, Daoyuan Jin, Scott MacGregor, Fatemeh Maghsood, Tian Qiu, Rye Weber, Jinhong Yu, and Guangxun Zhai. Footnotes Supplementary data to this article can be found online at https://doi.org/10.1016/j.plaphe.2026.100264. Contributor Information Changying Li, Email: cli2@ufl.edu. Yu Jiang, Email: yujiang@cornell.edu. Supplementary data The following is the Supplementary data to this article: Multimedia component 1 References Associated Data Supplementary Materials Multimedia component 1 Data Availability Statement The dataset can be accessed at: https://cornell.box.com/s/dh69xf84464yrc1vlws92l1tflx7qa89.