VT-SCC: Visual-text semantic alignment via multimodal guidance for unified staple crop organ counting Authors: Lanying Wang, Zhengbo Liu, Xinyu Dong, Hainan Wang, Wei Guo, Yuanyuan Xiao, Qi Wang Published: 2026 ABSTRACT AbstractAccurate counting of yield-related plant organs is fundamental to crop breeding and management. However, scalable counting remains challenging because crop morphology varies widely and field environments are complex. Current specialized models lack flexibility, whereas generic few-shot counters often exhibit feature ambiguity arising from their reliance on visual exemplars, leading to frequent misidentification in dense scenes. To address these challenges, we developed VT-SCC, a visual-text unified staple crop organ counting framework that incorporates semantic guidance into visual representation learning. VT-SCC integrates a Scale-Adaptive Feature Aggregator (SAFA), which uses a multi-branch structure to adaptively capture targets with large size variation, and a Semantic-Guided Channel Modulator (SGCM), which uses category text prompts to calibrate visual features dynamically and suppress background interference. In extensive experiments on a diverse dataset comprising wheat (Triticum aestivum), maize (Zea mays), rice (Oryza sativa), and soybean (Glycine max), VT-SCC achieved a mean absolute error(MAE) of 3.28 and a root mean square error(RMSE) of 4.48, outperforming existing generic counters while remaining competitive with specialized models. VT-SCC also generalized effectively in few-shot transfer learning tasks, adapting to unseen domains with minimal data. These observations demonstrate the feasibility of VT-SCC as a unified agricultural counter and highlight its potential as a scalable, robust solution for high-throughput phenotyping. The code and dataset are publicly available at http://vt-scc.samlab.cn.Keywords: Agricultural vision, Crop organ counting, High-throughput phenotyping, Multimodal prompts Introduction Wheat (Triticum aestivum), maize (Zea mays), rice (Oryza sativa), and soybean (Glycine max) provide approximately 60% of human caloric intake [1]. Ensuring the productivity of these staple crops through accelerated breeding is vital for global food security. With the global population projected to reach 9.7 billion by 2050 [2], the current rates of crop production are insufficient to meet future needs; global demand for crops increases at a rate of 2.4% annually while genetic yield gain averages only 1.3% [3]. To address this deficit, breeders have been focusing on optimizing yield components such as the number of reproductive organs per unit area [4]. However, population-scale trait screening using traditional manual sampling is labor-intensive and prone to human error [5,6]. While unmanned aerial vehicles (UAVs) and high-throughput phenotyping (HTP) platforms have been used to collect high-resolution imagery for improved screening efficiency [7], robust counting algorithms are required to translate these data into reliable phenotypic evaluations under complex field conditions. Deep learning methods have been extensively explored for automating field counting and fall into three categories: regression-based counting, instance segmentation, and object detection. Regression-based approaches are effective for processing dense scenes and reducing annotation requirements. Examples include WheatNet [8], the UAV-based density algorithm developed by Wu et al. [9], and CSNet [10]. While these methods are effective for automated field counting in crowded scenes, they lack individual bounding boxes, limiting the localization of specific plant organs. Instance segmentation methods aim to outline the boundaries of crop organs. Examples include AS-SwinT [11] and a tomato (Solanum lycopersicum) counting model proposed by Afonso et al. [12]. However, despite providing detailed shape information, the high cost of high pixel-level annotation hinders the scaling of these methods for large agricultural datasets. Object detection is frequently used for phenotyping because it balances the need for localization with low annotation costs. YOLO and Faster R-CNN are widely used due to their ability to balance accuracy and speed [13]. For specific applications, TasselLFANet [14] and the Panicle-Cloud platform [15] demonstrate superior performance in instance-level quantification. Despite these advancements, most existing models share a common limitation: they were designed for use with specific crops. For instance, wheat models often perform poorly for maize or soybean due to morphological differences. This species-specific dependency limits cross-species generalization, hindering the development of a unified counting paradigm for multiple crops. Few-shot counting (FSC) addresses the scalability bottlenecks of the "siloed paradigm" (Fig. 1A) by easing the constraints imposed by fixed-parameter models with static feature spaces (Fig. 1D (1)). FSC shifts the focus toward task-adaptive inference. To date, most agricultural studies using FSC have used the visual exemplar-based approach, which uses a few cropped images as reference patterns. For instance, Zhao et al. [16] and Ye et al. [17] used this approach to perform the multi-species tasks shown in Fig. 1B. More recently, TasselNetV4 [18] introduced a unified plant-counting model that counts classes across species through multi-branch box-aware counters and token-level local regression. However, while these methods reduce retraining requirements, they primarily rely on an "extract-and-match" mechanism using visual exemplars. Because visual exemplars often include irrelevant background elements, background noise can interfere with target features and create visual ambiguity, as illustrated in Fig. 1D(2), resulting in false positives. In addition, multi-granularity adaptation remains a persistent challenge. Because crop organs differ substantially in physical size and geometric structure, the model must process features at multiple levels. Paradigm comparison and motivation. A Siloed paradigm with limited cross-crop generalization. B Few-shot paradigm relying on visual exemplars and sensitive to background noise. C Our unified framework (VT-SCC) for major staple crop counting. D Micro-motivation showing the transition from (1) species-specific isolation caused by fixed parameters and domain gaps, through (2) visual-similarity entanglement and distractor activation within the visual similarity scope, to (3) text-guided multi-granularity alignment, in which the text-guided semantic field separates target features from distractors and other classes. To overcome these limitations, we developed VT-SCC, a visual-text framework for directable staple-crop counting (Fig. 1C). VT-SCC builds on Grounding DINO [19] while adapting it to the scale variation, morphological diversity, and background ambiguity of agricultural scenes. A Text-Guided Semantic Field (Fig. 1D(3)) aligns multi-granularity visual tokens into a shared representation, while two modules provide the key adaptations: the Scale-Adaptive Feature Aggregator harmonizes crop features across species and target sizes, and the Semantic-Guided Channel Modulator uses task instructions to suppress non-target interference. Together, these components move beyond prompt matching in multimodal counters such as CountGD [20], enabling robust counting of multiple staple crops within a single architecture. Our main contributions are as follows: •We introduce VT-SCC, a unified visual-text framework for directable counting across diverse staple crops, reducing reliance on crop-specific models. We introduce VT-SCC, a unified visual-text framework for directable counting across diverse staple crops, reducing reliance on crop-specific models. •We propose task-guided feature modulation and scale-adaptive aggregation to improve counting under variable crop size, morphology, and background clutter. We propose task-guided feature modulation and scale-adaptive aggregation to improve counting under variable crop size, morphology, and background clutter. •We build the M2-Crop dataset and show that VT-SCC matches specialized counters while generalizing well in few-shot transfer settings. We build the M2-Crop dataset and show that VT-SCC matches specialized counters while generalizing well in few-shot transfer settings. Results Experimental details The proposed VT-SCC framework was built on the Grounding DINO architecture. During training, the Swin Transformer backbone and BERT text encoder were frozen to maintain representational stability. The model was optimized using the AdamW algorithm with a global initial learning rate of 1e-4 and a weight decay of 1e-4. We used a StepLR scheduler to reduce the learning rate every 10 epochs over a total of 50 training epochs. The loss function was a weighted combination of Focal Loss and L1 Loss, synergistically optimizing target classification and bounding box regression. We used a multi-scale augmentation strategy for image processing during the training phase, dynamically scaling the shortest side of the input images between 480 and 800 pixels. By contrast, the architecture supports unrestricted input resolutions during the inference phase, allowing flexible processing of diverse images from various field sensors. Finally, we utilized the curated M2-Crop dataset for unified multi-species training, while reserving the MTC dataset for out-of-distribution evaluation to verify the model's generalization capability. Comparison with specialized methods on individual crops We conducted these experiments to establish a performance baseline and to verify the feature perception capability of the proposed architecture within specific crop domains. In this setting, the model operates as a crop-specific counter and undergoes individual training on datasets for wheat, soybean, rice, and maize. We benchmarked our method against a comprehensive set of baselines: general detectors such as Faster R-CNN [21], SSD [22], Mask R-CNN [23], FCOS [24], EfficientDet [25], and YOLO variants [[26], [27], [28], [29]]; density-regression models such as MCNN [30] and CSRNet [31]; and domain-optimized counters such as CSNet [10], WheatLFANet [32], TasselLFANet [14], YOLO POD [33], PoDNet [34], and SPCN [35]. This crop-wise evaluation isolated dataset-specific training effects to test whether the method consistently addressed shared phenotyping challenges, such as background ambiguity and instance occlusion. First, the proposed method achieved superior results for the GWHD 2021 wheat dataset, with an MAE (mean absolute error) of 3.38 (Fig. 2A). This performance was better than that of the domain-optimized CSNet [10] and general detectors such as Faster R-CNN [21], SSD [22] and DETR [36], as well as the transformer-based baseline TransCrowd-GAP [37]. This improvement indicates that the learned representation effectively suppresses confusion caused by complex foliage and textural similarities between wheat heads and backgrounds. Overall performance comparison across five representative datasets (MAE & RMSE). A–E Correspond to GWHD 2021, DRPD, MrMT, MTC and YOLO-POD, respectively. For DRPD, several baselines do not report MAE in their original papers (missing MAE bars). Bars with diagonal lines indicate our VT-SCC results. Second, our model reported an MAE of 2.65 and an RMSE of 4.12 for the YOLO POD dataset (Fig. 2E), which features dense soybean pod clusters with heavy overlap. The consistent advantage over dataset-oriented baselines such as YOLO POD [33], PoDNet [34], and SPCN [35], as well as general models including Mask R-CNN [23], Swin-S Mask R-CNN [38], FCOS [24], EfficientDet [25], and YOLO variants [[26], [27], [28], [29]], confirms that the proposed features remain separable under severe adhesion and occlusion. In addition, the VT-SCC framework demonstrated high reliability in small-target aerial scenarios using the DRPD dataset (Fig. 2B), achieving the lowest available MAE of 1.64. Compared to specialized models such as PlantBiCNet [17], P2PNet [39], TasselLFANet [14], Panicle-Cloud [15], and LKNet [40], our method maintained competitive accuracy without relying on strong assumptions tied to a particular viewpoint or repetitive background textures. Furthermore, using the MrMT dataset (Fig. 2C), our method achieved an MAE of 2.58, surpassing crop-oriented baselines such as TasselNetV3-Seg [41] and TasselLFANet-HE [14]. Despite substantial changes in maize appearance across developmental stages and interference from leaves, the lower MAE obtained using our method confirms that the representation remained consistent across morphological shifts, outperforming general detectors such as SSD [22], RetinaNet [42], CenterNet [43], and Yolov7-tiny [44]. Finally, we conducted a cross-domain evaluation of the unseen MTC dataset (Fig. 2D) to assess the robustness of the VT-SCC framework under a distribution shift and to determine whether the learned representations capture intrinsic morphological invariants. By training exclusively on the MrMT source domain and performing direct transfer inference on the target domain, our model achieved the lowest MAE of 4.84. This performance surpassed that of established baselines such as TasselNetV2 [45], TasselNetV2+ [45], BCNet [46], and CHCNet [47], confirming that the captured features remained informative despite variations in sensors, illumination, and background statistics, thereby validating the strong generalization capacity of our unified perception paradigm. Comparison with general models using staple crop datasets Building on the established perception capabilities of specialized models, we examined whether the unified parameters model trained on the M2-Crop dataset would possess superior robustness compared with general-purpose counters. This unified framework was benchmarked against five advanced general methods, namely CACViT [48], SAFECount [49], the vision-language model VLCounter [50], the multimodal model CountGD [20], and the state-of-the-art plant-specific foundation model TasselNetV4 [18], on a combined test set encompassing four distinct crop types. The results are listed in Table 1. Table 1Benchmark comparison on the M2-Crop dataset. Lower error metrics indicate better counting accuracy, and the best result in each metric is highlighted in bold.Table 1MethodMAERMSENRMSECACViT [48]4.129.5822.18%SAFECount [49]4.135.8513.54%VLCounter [50]3.695.6112.99%CountGD [20]3.914.8011.11%TasselNetV4 [18]3.364.9911.60%VT-SCC3.284.4810.37% Benchmark comparison on the M2-Crop dataset. Lower error metrics indicate better counting accuracy, and the best result in each metric is highlighted in bold. General exemplar-based models such as CACViT [48] and SAFECount [49] exhibit performance limitations when handling dense crop adhesion and complex field backgrounds. This gap reflects the inherent constraint of relying solely on visual prototypes, which are highly susceptible to feature entanglement caused by environmental interference. By contrast, VT-SCC injects semantic constraints via the SGCM to dissociate background noise from target identification at a foundational level, effectively mitigating the "visual ambiguity" challenge highlighted in the Introduction. We specifically evaluated our model against TasselNetV4 [18], a significantly advanced foundational model designed for plant counting. While TasselNetV4 demonstrated robust generalization across diverse species through its multi-branch architecture, our framework achieved superior results across all metrics listed in Table 1, including the MAE, which decreased from 3.36 (TasselNetV4) to 3.28. This improvement indicates that our multimodal semantic guidance provides more explicit task definitions than the purely vision-based "extract-and-match" paradigm. Finally, our framework achieved an error reduction of over 10% compared to the semantic-capable VLCounter [50], revealing the purification effect of the SAFA module on the feature space. Ablation studies Impact of prompt modalities To deconstruct the contributions of different prompt modalities and validate their synergistic effects, we conducted an ablation study evaluating the Text-only, Exemplar-only, and Combined configurations. This experiment compared the MAE and RMSE across these input modes on the M2-Crop dataset to reveal the complementary relationship between semantic constraints and visual prototypes in the feature reshaping process. The results are shown in Table 2(A). Table 2Ablation study results on the M2-Crop dataset. Lower MAE and RMSE indicate better counting accuracy; the best-performing variants are highlighted in bold.Table 2CategoryConfigurationMAE(↓)RMSE(↓)(A) Prompt modalityText-only3.464.79Exemplar-only4.136.02Combined (Text + Exemplar)3.284.48(B) Encoding methodOne-hot3.795.84Learnable4.236.64BERT3.464.79(C) Architectural moduleBaseline3.914.80Baseline + SAFA3.424.67Baseline + SGCM3.354.56Baseline + SAFA + SGCM3.284.48 Ablation study results on the M2-Crop dataset. Lower MAE and RMSE indicate better counting accuracy; the best-performing variants are highlighted in bold. First, the Text-only configuration yielded significantly lower error rates (both MAE and RMSE) compared to the Exemplar-only baseline. This superiority reflects the fundamental role of the semantic field in filtering background interference within complex agricultural environments. By injecting category semantics via the SGCM, the model constructs a high-level discriminative filter that effectively suppresses distractors with similar textures but no semantic relevance. The Exemplar-only mode exhibited higher error rates, as it relies on a visual "extract-and-match" paradigm. This approach is more susceptible to feature entanglement when targets and background elements are visually ambiguous, leading to increased prediction variance. The Combined configuration achieved the optimal performance across all metrics, with the MAE and RMSE further decreasing to 3.28 and 4.48, respectively. This improvement validates the synergistic mechanism in which text prompts define the broad semantic scope while visual exemplars provide concrete benchmarks to refine instance-level localization. This multimodal coordination ensures that the VT-SCC model maintains both discriminative robustness and high counting precision for morphologically heterogeneous crops. Analysis of category-encoding strategies To evaluate the need for multimodal semantic guidance and to determine whether pre-trained textual embeddings provide unique advantages for the SGCM module, we conducted a comparative experiment involving three encoding strategies: One-hot Encoding, which assigns fixed 4-dimensional IDs to the crop categories; Learnable Embedding, which utilizes a randomly initialized layer to learn representations during training; and BERT Embedding, which leverages pre-trained semantic priors to generate high-dimensional feature vectors. The results using the M2-Crop dataset are summarized in Table 2(B). BERT Embedding achieved the lowest MAE and RMSE, outperforming the non-semantic encoding methods. This result suggests that pre-trained linguistic knowledge serves as a stable semantic anchor for the SGCM, helping the model distinguish target organs from visually similar background clutter. Learnable Embedding exhibited the highest error rates across all metrics, indicating that without the structural constraints of linguistic knowledge, the Prompt Query (Q) suffers from optimization instability during training. In the absence of semantic priors, the learnable parameters fluctuate significantly, making it difficult for the modulator to consistently activate target-relevant feature channels in dense field environments. In addition, One-hot Encoding provided a lower error rate than Learnable Embedding because its fixed, orthogonal IDs offer a rigid task definition that prevents optimization drift. However, its error rate was worse than that of the BERT-based approach, as it lacks the semantic relationships needed to handle the visual ambiguity inherent in agricultural scenes. In summary, these results confirm that BERT embeddings are not redundant but function as essential semantic filters, bridging the gap between low-level visual patterns and high-level counting intentions across diverse crops. Contributions of the proposed modules To isolate the contributions of our proposed modules from the pre-trained foundation, we first established the performance ceiling of the adapted backbone. Deploying Grounding DINO without task-specific adaptation yielded an MAE of 41.93, confirming that generic multimodal weights cannot inherently bridge the agricultural domain gap. We then adopted a strong baseline that fully adapted the Grounding DINO architecture for counting under the frozen-backbone protocol described in Section 2.1. This baseline achieved an MAE of 3.91 (Table 2(C)), representing the maximum performance attributable to architecture adaptation and task alignment alone. The further reduction to 3.28 reflects the incremental gain produced by the SAFA and SGCM modules. To quantify the individual and collective contributions of the SAFA and SGCM modules, we conducted a stepwise ablation study on the unified test set. We evaluated the counting accuracy of the baseline, individual modules, and the integrated framework to verify the collaborative logic of the core components in addressing multi-granularity and visual ambiguity. The results are shown in Table 2C. First, incorporating the SAFA module into the baseline reduced the MAE by 12.5%, validating its capability to handle multi-granularity challenges arising from diverse morphological structures and harmonize heterogeneous features. This improvement enhances perceptual robustness across varying scales, constructing a more discriminative feature space at the structural level. Subsequently, adding only the SGCM module reduced the MAE to 3.35, confirming the efficacy of channel-wise semantic modulation in suppressing environmental interference. This improvement reflects SGCM's ability to construct a dynamic semantic field via text instructions, fundamentally mitigating the challenges related to visual ambiguity. In addition, the full configuration achieved optimal performance, demonstrating the complementarity between structural representation refinement and semantic guidance. This synergy stems from a hierarchical optimization process in which the SAFA refines the feature representations, and the SGCM precisely directs attention to ensure stability across heterogeneous morphologies. In summary, the experiment confirmed that integrating the SAFA and SGCM modules is essential for high-performance counting. The unified VT-SCC framework therefore shows greater robustness and representational depth than traditional single-structure architectures in complex agricultural scenarios. Few-shot transfer experiments We conducted few-shot transfer learning experiments to assess the adaptability of the proposed framework under data-scarce conditions and domain shifts. The evaluation followed a two-stage protocol: pre-training on the generic FSC-147 dataset followed by fine-tuning on a minimal subset of the target agricultural dataset. Specifically, a standardized 3-shot protocol was implemented, in where three support samples per crop category were used for target domain adaptation. To rigorously evaluate the framework under non-idealized real-world conditions, these support samples were acquired via a random selection strategy from the target test distribution, bypassing any manual pre-screening or morphological intervention. Each randomly selected support instance was annotated with human-verified bounding boxes to serve as visual exemplars. During the few-shot inference phase, the confidence gating threshold τ was strictly set to 0.23 to align with the primary evaluation protocol. Cross-modality task guidance was consistently provided through predefined text prompts to ensure stable semantic field calibration under data-scarce conditions. We benchmarked our framework against various few-shot counters, including CounTR [51], SAFECount [49], CACViT [48], and VLCounter [50], with the data distribution and performance results shown in Fig. 3. Few-shot transfer learning evaluation under a 3-shot configuration. A Fine-tuning data distribution. B Performance comparison confirms the framework's adaptability to low-resource scenarios by leveraging architectural refinement to overcome data scarcity. First, quantitative analysis revealed that generic few-shot models encountered performance limitations when handling field-based agricultural scenes, with CounTR [51] and CACViT [48] yielding MAE values of 14.87 and 10.51, respectively. This gap stems from their reliance on visual similarity-based matching, which shows reduced effectiveness when addressing the dense patterns and complex backgrounds of agricultural environments during cross-domain migration. Subsequently, the full model further improved upon the CountGD baseline [20], reducing the MAE to 6.30 and RMSE to 8.86. This gain is attributed to the scale coordination by the SAFA module and the semantic supplementation by the SGCM module, confirming that architectural refinement can effectively compensate for the scarcity of visual reference information. In summary, the experimental results validate the feasibility of using the proposed framework for few-shot transfer tasks. This robust performance in low-resource scenarios demonstrates that the paradigm can effectively alleviate dependency on large-scale task-specific annotations, revealing its generalization robustness in cross-scenario counting of diverse agricultural targets. Regression analysis and model consistency Beyond average error metrics, we evaluated the linearity and goodness of fit between predicted counts and ground-truth values using regression analysis. We used the coefficient of determination R2 as the primary metric to quantify how well the model explained the variance in the target data. Fig. 4 shows a comparative scatter plot analysis between the specialized models trained on single crops and the unified model. Regression analysis. A-D Specialized models. E-H Unified model. I Overall performance. High linearity confirms the unified paradigm maintains global consistency by capturing universal features across species. First, specialized models trained on individual crop subsets established a high baseline for prediction linearity, with all R2 values exceeding 0.95. This indicates that under a task-specific training mode, the architecture effectively captured the morphological features of the target crops. Subsequently, the results for the unified model demonstrate that joint training preserves or even enhances prediction consistency across most species. This gain stems from multi-task complementarity: by sharing morphological constraints across species, the model filters dataset-specific biases to focus on universal geometric cues. Furthermore, the unified model achieved an aggregate R2 of 0.9755 across the combined test set. In summary, the results of the regression analysis validate the notion that VT-SCC possesses strong global prediction consistency. This high-consistency fitting performance across varied scenarios confirms that the unified perception paradigm achieves robust inference for diverse agricultural targets without sacrificing local precision. Qualitative analysis of counting results Feature-response visualization across crops We qualitatively evaluated the performance of VT-SCC in diverse agricultural scenarios using predicted distributions and attention heatmaps. Fig. 5A shows the original images, model predictions, and corresponding feature responses for four crops, indicating that the architecture remains robust under occlusion, adhesion, and morphological variations. Qualitative analysis of the unified counting framework. A Feature-response and localization visualization across diverse crops highlights precise localization and consistent responses under occlusion and morphological variations. B Qualitative comparison in high-density scenes demonstrates that VT-SCC remains stable and accurate against heavy overlap and ambiguity compared to general models such as TasselNetV4, SAFECount, CACViT, and VLCounter. Analysis of soybean, wheat, and rice showed that the targets often adhered to surrounding structures and partially occluded each other, making them visually similar to the background foliage. The attention heatmaps are concentrated on the core target regions and suppress background responses, suggesting that the model extracts target-related cues under clutter and occlusion conditions. Analysis of maize demonstrated the adaptability of VT-SCC to distinct target geometries. For the branching structure of maize tassels, the response regions align with the physical contours and cover the radiating branches, indicating that the unified architecture adjusts its focus according to the structure of the target. In summary, Fig. 5A indicates that the framework achieved reliable localization and discrimination in complex field environments and preserved cross-crop representational consistency, supporting the robustness of VT-SCC for diverse agricultural targets. Qualitative results in crowded and occluded scenes To further examine the behavior of VT-SCC under severe crowding, we conducted an additional qualitative visualization on high-density scenes with heavy overlap and compared VT-SCC with TasselNetV4 [18], SAFECount [49], CACViT [48], and VLCounter [50]. The results are shown in Fig. 5B. The results obtained with VT-SCC remained close to the ground truth across all dense scenarios, indicating that this tool maintained reliable performance under severe occlusion and adhesion. The generic counters tended to undercount in crowded clusters, suggesting that purely visual cues are often insufficient to separate individuals when plant parts appear to merge. TasselNetV4 and SAFECount performed better than the weakest models, but their predictions exhibited greater variability under similarly dense conditions, reflecting limited robustness to overlap-driven ambiguity. By contrast, VT-SCC preserved stable predictions across these dense scenarios, which aligns with our design: the SAFA module supports scale-robust feature aggregation, the SGCM uses text instructions to recalibrate channels and suppress non-target interference, and the cross-modality decoder matches candidate regions to task instructions for counting. Finally, in the extremely crowded scenario, VT-SCC closely matched the ground truth, whereas generic counters showed marked deviations. These results further support the effectiveness of VT-SCC in dense field environments. Overall, the results of the two visual studies complement each other: Fig. 5A provides feature-response evidence for robust localization across crops, and Fig. 5B provides evidence that VT-SCC remains stable in high-density overlap scenarios. Discussion In this study, we developed VT-SCC, a unified framework designed to overcome the scalability limitations of specialized models for crop organ counting. This tool integrates SAFA and SGCM to process features across crops including wheat, maize, rice, and soybean. This framework captures cross-species visual invariants and adapts to morphologies ranging from node-distributed pods to branching tassels without the need for retraining. VT-SCC achieved a mean absolute error (MAE) of 3.28, demonstrating its potential for automated phenotyping. VT-SCC demonstrates robustness in field backgrounds due to its use of semantic constraints to resolve visual ambiguities. The optimization between structural refinement and semantic alignment ensures strong performance across different stages of plant growth. This framework uses a text-guided deep alignment strategy instead of background segmentation because target organs and foliage often share identical spectral and textural characteristics. Due to these similarities, segmentation models can fail to define geometric boundaries. Furthermore, segmentation-based approaches require pixel-level inference and annotation costs, whereas this framework uses category-specific text prompts as a cognitive prior to filter background interference based on identity. The effectiveness of the SGCM relies on the semantic alignment between text prompts and target features. In high-density environments, the disambiguation capability depends on the specificity of the task instructions. The M2-Crop dataset generated in this study serves as a bridge between generic few-shot counting and specialized agricultural tasks. M2-Crop encompasses variations in target granularity, sensor altitude, and illumination conditions. Specifically, as shown by statistical analysis Supplementary Fig. S2B), the Nearest Neighbor Distance (NND) distribution of the dataset encompasses a wide range, from dense clusters to sparsely distributed targets. This variance in spatial density mitigates the risk of overfitting to a single density pattern and functions synergistically with the SAFA. The transition from dense to sparse features provides diverse scenarios that train the dynamic adjustment of multi-scale receptive fields. By adapting to varying levels of crowdedness, this statistical diversity guides models toward learning universal geometric representations rather than relying on textural memorization. This could explain why the framework effectively generalizes to unseen domains and maintains high counting precision, provided that the target distribution falls within the learned density range. The inclusion of multimodal annotations encompassing text, exemplars, and points supports the development of interactive counting paradigms. We plan to expand the framework for use with additional crop species and cultivation environments. Materials and methods Previously constructed datasets and new dataset construction Traditional agricultural counting datasets usually focus on a single species. The fragmented nature of these datasets hinders the development of scalable cross-species counting models. Moreover, developing four independent datasets requires repetitive data collection, cleaning, and validation workflows, which significantly increase the annotation burden. The primary motivation for constructing a multimodal, staple crop benchmark is to bridge these isolated data domains and enable the model to learn universal visual invariants across different crops. However, the construction process involves several significant technical challenges. First, standardizing heterogeneous data requires a complex workflow to unify multiple annotation formats into a single detection setting. Second, establishing multimodal pairs requires generating and verifying category-specific text prompts to provide accurate semantic guidance. Third, ensuring data quality across disparate sources requires rigorous curation and screening to exclude low-quality samples that contain insufficient target instances. Finally, supplementing bounding box annotations for datasets that contain point labels to support cross-domain evaluation requires a laborious manual effort that is essential to the construction workflow. We constructed the M2-Crop dataset, also known as the Multimodal and Staple Crop Counting dataset. This benchmark integrates four representative datasets: GWHD 2021 [52] for wheat, MrMT [14] for maize, DRPD [15] for rice, and YOLO POD [33] for soybean. To address inconsistent data structures, we unified heterogeneous labels from the original sources, including XML, JSON, and point-only labels, into a standardized detection schema with bounding box coordinates. For the multimodal configuration, we associated images of wheat, maize, rice, and soybean with the text prompts “Wheat head.”, “Maize tassel.”, “Rice panicle.”, and “Soybean pod.”, respectively. Simultaneously, we randomly sampled three ground-truth bounding boxes per image as visual exemplars, while the remaining targets were annotated with center points using fixed-size pseudo-boxes. Following this curation, the M2-Crop dataset comprised 13,474 images, which were partitioned into training, validation, and test sets at an approximate ratio of 8:1:1. As shown in Supplementary Fig. S1, M2-Crop encapsulates the intrinsic challenges of field phenotyping, such as extreme scale variations and heavy occlusion. Additionally, the MTC [53] dataset for maize is reserved exclusively for cross-domain testing. We manually added the required bounding box annotations for visual exemplars across all images in this dataset to evaluate the model's generalization to previously unseen data distributions. Dataset unification and multimodal configuration Supplementary Table S3, compares the technical attributes of the proposed M2-Crop dataset with several established plant-counting benchmarks. Specialized datasets, such as GWHD 2021 [52] and YOLO POD [33], provide essential high-quality annotations for specific crops in a single domain. The construction of the PAC-105 and PAC-Somalia datasets [18] represents a notable advancement in enabling cross-species counting. Specifically, PAC-105 encompasses 105 categories across 64 plant species, while PAC-Somalia includes 32 unique species native to Somalia, collectively establishing a data foundation for class-agnostic plant counting. While vision-based studies have achieved broad species coverage, M2-Crop was developed to address the demand for greater informational depth for core staple crops. Within its target domains, M2-Crop provides an extensive collection of 13,474 images, establishing a dense cross-species data foundation for robust feature learning. As highlighted in Supplementary Table S3, a key technical distinction is the transition from “visual-only exemplars” to multimodal support. Unlike earlier approaches that rely on visual similarity, M2-Crop integrates “Box + Text” annotations, enabling models to leverage category-specific semantic guidance to resolve visual ambiguities common in complex field environments. By adopting a multi-task detection paradigm, the dataset provides the structured support required for unified frameworks that combine precise spatial localization with deep semantic alignment. VT-SCC framework architecture We developed a unified staple crop counting framework, as shown in Fig. 6A. The Swin Transformer backbone extracts multi-scale image representations, whereas the Scale-Adaptive Feature Aggregator (SAFA) dynamically recalibrates these feature channels to adapt to the intra-class scale variance of different crops and reduce scale-confused responses. Simultaneously, text descriptions are encoded into semantic embeddings using a BERT model, while visual exemplars are transformed into visual tokens via RoI Align to collectively provide explicit task definitions. To resolve semantic ambiguity, these multimodal prompts guide the SGCM, which functions as an alignment mechanism to selectively activate the feature channels that correspond to the specific counting target. Finally, the cross-modal decoder regresses the aligned features into bounding boxes to generate an accurate count. In the following subsections, we discuss the implementation principles of each part of the framework in detail. Overview of the VT-SCC framework and core modules. A Overall Architecture: The framework fuses a SAFA-enhanced visual stream with a multimodal prompt stream (text and exemplars) via a Deep Alignment Encoder. A Cross-Modality Decoder then processes these aligned features to output the final count yˆ. B SAFA: This module disentangles multi-scale features through a Decoupler and refines target responses using a weight-based Calibrator. C SGCM: This module performs semantic filtering by constructing a Channel Affinity Matrix to suppress background noise. Scale-adaptive target perception A significant challenge in unified staple crop organ counting is the intra-class scale variance caused by diverse crop genotypes and fluctuating imaging distances. The Swin Transformer backbone extracts raw visual representations, but directly feeding these features into the subsequent alignment module is suboptimal because they often contain scale-confused noise. To address this issue, we developed the SAFA, which serves as a structural bridge between the backbone and the transformer encoder. This module refines the raw output into scale-robust representations before the deep alignment stage. As illustrated in Fig. 6B, the SAFA consists of two cascaded components: the Decoupler and the Calibrator. The process begins with the Decoupler, which reduces scale ambiguity through a parallel multi-branch architecture that disentangles visual features using heterogeneous convolutional kernels where k denotes the convolution kernel size in pixels and takes values of 1, 3, 5, and 7. The fine-scale branches focus on extracting local textural cues vital for distinguishing compact crop targets, while branches with larger kernels capture broader morphological contexts to preserve the structural integrity of spatially extensive targets. The outputs from these branches are concatenated to form the multi-scale channel stack Xdecoupled as defined in Equation (1): where B denotes Batch Normalization, and ϕ is the ReLU activation function. To prevent redundancy from irrelevant scales, the Calibrator functions as a soft-selection mechanism. This component compresses the spatial information of the multi-scale stack into a channel descriptor Z via global average pooling (GAP). The bottleneck multi-layer perceptron (MLP) then calculates a scale-aware weight vector S to encode the importance of each feature scale for the current image. This vector reweights the decoupled features to suppress noise and highlight target-specific responses. The final scale-adaptive feature Fenhanced is obtained via a residual connection as shown in the following equations: This refined feature Fenhanced serves as the purified visual foundation for the entire framework. By resolving scale mismatches early in the encoding stage, the SAFA ensures that the downstream transformer receives a robust representation, which facilitates precise semantic alignment in the subsequent steps. Visual-text instruction embedding To address the inherent challenges of morphological diversity and visual ambiguity in staple crop environments, the proposed framework uses a multimodal prompt mechanism to define counting tasks. Different crop organs, such as branching maize tassels and clustered soybean pods, exhibit significant structural variations that are difficult to capture using a single modality. As illustrated in the prompt stream in Fig. 6A, text descriptions and visual exemplars are independently encoded and subsequently concatenated to provide both global semantic guidance and local appearance references. First, the natural language description T is encoded by a frozen BERT model to derive the semantic category embedding Ftext. This semantic field provides a high-level definition of the target category, helping the model distinguish crop organs from visually similar background elements such as foliage and stems. Simultaneously, to capture the specific instance-level morphological details of the current scene, RoI Align is used to extract visual tokens Fvisual from the SAFA-enhanced features Fenhanced based on exemplar boxes B. These visual signals provide concrete scale and texture benchmarks that compensate for the abstraction of text prompts. These two modalities are concatenated along the sequence dimension to form the final task-guidance prompt Fprompt: This unified embedding strategy ensures that the model effectively bridges the gap between abstract task definitions and complex physical appearances, allowing for robust counting across diverse agricultural targets. Visual-text deep alignment The SAFA module establishes a robust representation that adapts to varying target sizes. However, precise counting in complex field environments requires more than just size adaptation; robust discriminative power is needed to distinguish target organs from background elements with highly similar spectral or textural characteristics. In dense agricultural scenes, the visual signals of target organs are often entangled with those of foliage and stems, creating semantic ambiguity that cannot be resolved by representations focused primarily on geometric scale. To provide the model with “directable” cognitive capabilities and bridge the gap between low-level visual patterns and high-level counting intentions, we developed a visual-text deep alignment framework. This framework functions as a task-specific semantic filter, ensuring that the model selectively focuses on the intended crop category while suppressing interference from visually similar distractors. As illustrated in Fig. 6C, the alignment process begins with a dual-stream interaction mechanism. The visual tokens Fenhanced and prompt tokens Fprompt undergo information exchange via Bidirectional Attention (Bi-Attention) to establish a mutual multimodal context, yielding the updated features Fenhanced′ and Fprompt′. To achieve the high semantic purity required to distinguish target organs from cluttered backgrounds, the prompt tokens are aggregated through a Self-Attention module to generate a refined Prompt Query (Qp). Building on this foundation, the Semantic-Guided Channel Modulator (SGCM) executes core semantic recalibration. By compressing the global spatial statistics of the visual features into a Global Visual Key (K) via GAP, the framework computes the interaction between Q and K to derive the Channel Affinity Matrix (A): where C denotes the channel dimension used for scaling to stabilize gradients. This affinity matrix subsequently functions as a dynamic semantic filter to modulate the visual value (V) corresponding to the enhanced visual features. This operation selectively activates feature channels containing target-related semantics while suppressing those dominated by background noise. To ensure the stability of the feature manifold and to achieve smooth semantic injection, a learnable gating parameter γ controls the modulation intensity, generating the aligned features Faligned via a residual connection: Finally, these semantically refined features are forwarded to the Deformable Attention module. Guided by the clear semantic instructions from the SGCM, this mechanism accurately samples key points across spatial dimensions to handle the irregular geometric deformations inherent in crops. This step completes the transition from semantic alignment to precise spatial localization, priming the features for subsequent decoding and counting regression. Cross-modal query decoding and instance-aware counting Generating reliable phenotypic data requires the precise localization of individual crop organs, yet this process is often hindered by the extreme density and heavy occlusion inherent in field environments. In typical agricultural scenarios, such as clustered soybean pods or overlapping wheat heads, the visual boundaries between targets become blurred, making it difficult for traditional regression-based models to resolve entangled instances into discrete individuals. This lack of instance-level resolution often leads to inaccurate phenotypic evaluations when dealing with complex crop morphologies. To resolve these entangled targets, the decoding stage translates semantically aligned representations into discrete physical entities through a query-based mechanism. This stage uses a set of learnable query embeddings Q to serve as semantic anchors, which aggregate key visual features from Fenhanced″ using Multi-Scale Deformable Cross-Attention (MS-DCA). In the l-th decoding layer, each query hi dynamically samples response regions based on the bounding box bˆi(l−1) predicted from the previous layer. The state update is formulated as: where hi′ denotes the query feature after self-attention, and P(·) represents the reference point projection. By mapping vision-language consistent features onto specific bounding boxes, the framework shifts from abstract global counting to grounded instance-aware localization. The final query features H(L) are forwarded to prediction heads where a contrastive learning mechanism calculates the classification alignment score matrix S. Using the updated multimodal tokens Fprompt″ as the semantic prototype, the score is generated as: Finally, the count estimate yˆ is derived by filtering the detection set through a confidence gating strategy: This instance-aware paradigm ensures that the final count is substantiated by interpretable spatial evidence, satisfying the requirement for high precision in high-throughput phenotyping and trait screening. Evaluation metrics To quantitatively evaluate the counting performance of the unified framework across diverse agricultural scenarios, we defined the final predicted count yˆ as the total number of valid bounding boxes exceeding a confidence threshold τ during inference. We adopted four metrics for assessment: Mean Absolute Error (MAE), Root Mean Square Error (RMSE), the Coefficient of Determination (R2), and the Normalized Root Mean Square Error (NRMSE). Specifically, the MAE intuitively reflects the average magnitude of deviation between predicted and ground-truth counts. The RMSE serves as a measure of robustness, particularly in extreme counting cases, by penalizing larger prediction errors more heavily. R2 is utilized to assess the goodness of linear fit between the predicted distribution and the ground truth, where a value closer to 1 indicates superior explanatory power. Furthermore, the NRMSE provides a scale-independent assessment of accuracy across diverse crop types with varying target densities. By normalizing the error against the population mean, the NRMSE facilitates a fair performance comparison between high-density and low-density scenarios. The specific calculation methods are shown in Eq. (11) to Eq. (14). These metrics are formulated as follows: where N denotes the total number of test samples, yi represents the ground truth count for the i-th sample, yˆi is the corresponding predicted count, and y¯ represents the mean of all ground truth counts. CRediT authorship contribution statement Lanying Wang: Writing – review & editing, Writing – original draft, Visualization, Validation, Supervision, Methodology, Investigation, Formal analysis, Data curation, Conceptualization. Zhengbo Liu: Writing – original draft, Methodology, Investigation. Xinyu Dong: Visualization, Validation, Methodology, Investigation. Hainan Wang: Methodology. Wei Guo: Writing – review & editing, Data curation. Yuanyuan Xiao: Writing – review & editing, Writing – original draft. Qi Wang: Writing – review & editing, Writing – original draft, Conceptualization. Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.